mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-10-14 16:18:10 +00:00
tiennm99
d60c81e5a6
* refactor(parser): define one canonical student schema for all datasets
The 2016 and 2017 parsers were the same crate with divergent SQL. DDL, INSERT
and the subject regex table lived in each dataset's TOML config, so four copies
had to be kept in step by hand — which is how the two schemas drifted apart.
Move all of it into src/schema.rs as a single 22-column definition: 6 identity
columns (adding ten_cum_thi and gioi_tinh, previously 2016-only) and 16 subject
columns (the union of both exam years). Columns a dataset carries no data for
bind NULL, costing ~1 byte per row.
This collapses writer.rs to one insert path: insert_row_2016, SCORE_FIELDS_2016,
SCORE_FIELDS_2017 and the SCORE_FIELDS alias all go away. Configs shrink to the
parse rules that genuinely vary per dataset — sheet mode, column indices, SBD
validation, header tokens, blank-row stripping — and carry no SQL at all.
Config parsing now denies unknown fields, so a leftover [schema] block fails
loudly instead of looking effective while schema.rs drives the build.
idx_ten_cum_thi is partial, so it holds zero entries on the three 2017 datasets
where the column is always NULL.
Adds db-stats.js and verify-parity.js to prove no data moved: row counts,
per-column non-NULL counts and a deterministic student sample are compared
against databases built from the previous code.
Verified across all four datasets — row counts, every pre-existing column count,
and all sampled students are identical. Database size grows 1.4-2.2%.
* style(parser): clear pre-existing clippy warnings
cargo clippy --all-targets -- -D warnings failed with 8 errors before the schema
refactor and was never part of CI. Clearing them so the gate is meaningful from
here on.
All behaviour-preserving: slice contains() over iter().any(), is_multiple_of()
over a modulo, a needless borrow in a test helper, and a doc-list indent.
process_mapped_row keeps its 8 parameters behind an allow attribute — the
signature mirrors the JS column map one-for-one, and bundling the indices into
a struct would hide that correspondence.
Kept separate from the schema change so that diff stays readable.
* refactor: collapse the two projects into one tree and move to npm
The repo held two near-duplicate projects. 2016/ and 2017/ each carried their
own React frontend, their own copy of the same Rust crate, and their own
package manager setup. 2016/tools/sync-from-thptqg2017.sh existed purely to
copy the parser source between them.
New layout:
index.html + src/ the 2017 frontend, now the only one
data/<id>/ 2016, 2017, 2017-old, 2017-old2
parser/ the single Rust crate, configs renamed to <id>.toml
docs/ both projects' docs, 2016 copies suffixed -2016-legacy
pending the merge pass
<id> is now one identifier end to end: data/<id>/ feeds parser/configs/<id>.toml
and produces db/<id>.db.gz.
pnpm gives way to npm. pnpm-workspace.yaml existed only to whitelist
better-sqlite3's native build, which npm permits by default, so it has no
equivalent and is simply gone. Lockfiles cannot be converted; package-lock.json
is generated fresh. The migration direction is safe — pnpm's strict layout
forbids phantom dependencies, so anything that resolved under pnpm resolves
under npm's flat tree.
Adds parser/scripts/build-db.js and src/datasets.js: the four dataset IDs are
declared once and read by both the build tooling and (from the next phase) the
frontend.
Follow-on fixes the move made necessary:
- eslint's Node-globals override pointed at scripts/, now parser/scripts/
- crawl-baotintuc.js wrote to <root>/data, now data/2017
- golden tests loaded configs by their old thptqg*-data.toml names
Drops the #[ignore]d Rust-vs-Node golden test. It shelled out to pnpm to run
scripts/build-database.js, a file removed when the parser was ported to Rust,
so it could never pass. check-duplicates.js and diff-datasets.js were already
broken before this change and are annotated as such rather than half-fixed.
63 Rust tests pass and clippy is clean from the new location.
* feat(frontend): serve all four datasets and the hub from one app
The 2017 frontend becomes the only frontend. It now resolves which dataset to
show from the URL, so the four dataset views and the /thptqg/ landing page all
come from a single build instead of four builds plus a hand-written static
index.html.
Routing is an exact match on one path segment, because the segment is the
dataset id: /thptqg/2017-old/ -> "2017-old". The nested form this replaces
(/thptqg/2017/old/) would have needed longest-prefix matching, since it also
starts with /thptqg/2017/. Previously published nested URLs are rewritten to
their flat equivalent with the query string intact, so ?q= deep links keep
working.
Restores what the 2016 site had and the 2017 site lacked:
- Cụm thi and Giới tính columns, in the table and on the detail card
- exam-ID lookup for letter-prefixed số báo danh
That second one would have been a severe regression. 616,593 of 877,461
candidates in the 2016 dataset — 70.3% — have an SBD like BAL000001, and the
2017 app matched /^\d+$/ only, so exam-ID search would have silently failed for
most of that year. Detection now lives in lib/query-mode.js, shared by App and
SearchForm, which had already drifted apart on exactly this rule. Letter
prefixes are upper-cased before lookup so "bal000001" resolves.
Neither dataset needs a conditional in a component. Columns where every row is
NULL are hidden, so 2016 rows surface Cụm thi / Đức / Nhật while 2017 rows
surface KHTN / KHXH / GDCD / Nga. computeBlocks() likewise skips blocks with a
missing subject, so one admission-block list covers both years — D05 and D06
are added now that German and Japanese scores exist.
Per-dataset content moves into the registry: title, source, database size,
search examples and SQL presets. The 2016 presets are lifted from the deleted
2016 component; the Long An (49xxx) queries stay on 2017 only, where that SBD
prefix means something.
The subject list, previously maintained separately in score-table.jsx and
student-detail.jsx, is now declared once in lib/subjects.js mirroring
SCORE_FIELDS in the parser.
* style(frontend): document the four one-shot setState-in-effect sites
eslint-plugin-react-hooks v7 flags setState inside an effect body. All four
occurrences predate this refactor and are one-shot initialisation or external
sync, not the cascading-render pattern the rule targets:
- hydrating a ?q= deep link once the database is ready
- reading the candidate count for the footer
- auto-running the schema preset when the SQL tab first opens
- mirroring the parent-owned query into SearchForm for deep links and clear
Each gets a scoped disable with a comment explaining why it is correct here,
rather than a blanket rule change. Restructuring these properly would mean
reworking deep-link hydration and auto-search with no test harness to catch a
regression; that is worth doing separately, not inside this refactor.
npm run lint is now clean.
* build: produce the whole site from one Vite build
The four build variants existed only because the app could not resolve its own
dataset. Now that it can, vite.config.js is a single build with an absolute
base and no VARIANT switching, and the workflow compiles Rust once, installs
Node dependencies once, and builds the site once — it previously built the same
Rust crate twice and ran two separate pnpm installs.
Because base is absolute, the emitted index.html references /thptqg/assets/...
regardless of where it is served from, so the same file works as an entry point
at any depth. scripts/assemble-site.js copies it to each dataset path and to
the two legacy nested URLs, giving a real static file at every published route.
That is what removes the need for an SPA 404-fallback redirect, which would
otherwise have rewritten URLs and interfered with the ?q= deep links.
Generated databases move to a gitignored .build/public, which Vite consumes as
its publicDir. build-db.js gzips without -k, and the assemble step then refuses
to finish if any uncompressed database artefact reached the output — .db,
.db-journal, .db-wal or .db-shm. Previously a raw 100+ MB database was written
into the source tree and deleted afterwards by an rm in the workflow, so
shipping one was a missing cleanup step away.
Both build scripts import DATASET_IDS from src/datasets.js rather than
repeating the dataset list in workflow shell, so the four ids are declared in
exactly one place across the frontend, the database build and the assembly.
Verified by running the full pipeline locally and serving the artifact over
HTTP: all nine routes return 200, every entry point is byte-identical, asset
references are absolute, and the databases are fetchable. The guard was tested
by injecting a raw .db and .db-journal into the build, which failed the
assembly as intended.
* docs: rewrite for the unified repo and record the parity gate
The docs still described two standalone projects with separate frontends,
separate parsers and three Vite variants. Rewritten around what the repo now
is, merging both projects' copies rather than keeping one and discarding the
other — deployment-guide.md and system-architecture.md existed in both and
documented different pipelines.
project-overview.md goal, scope, constraints, the four datasets, history
system-architecture.md data flow, canonical schema, routing, how one
frontend serves both exam years without branching
data-pipeline.md per-dataset Excel formats, the three 2016 layouts,
overflow-sheet gotcha, expected row counts
deployment-guide.md the single-build workflow, adding a dataset,
why no uncompressed database can ship
Records the release gate in plans/reports/parser-parity-result.md: row counts,
all 18 pre-existing per-column non-NULL counts and the deterministic student
samples are identical across all four datasets, against databases decompressed
from the exact bytes the pipeline publishes. The 1,691 recovered
foreign-language scores are documented with the evidence they are real.
Also drops a machine-specific absolute path from a comment in
format_detect_2016.rs. The build-database.js citations there are kept: that
file no longer exists in this repo, but the references explain why several
parsing rules look arbitrary.
thptqg
Tra cứu điểm thi THPT Quốc gia — exam-score lookup for Vietnam's national high
school graduation exam. Client-side SQL (sql.js) over a SQLite database built
from the ministry's raw .xls score files by the Rust xlsxread parser.
Live at tiennm99.github.io/thptqg.
| Dataset | Exam | Candidates | Site |
|---|---|---|---|
2016 |
2016 | 877,461 | /2016/ |
2017 |
2017 | 861,068 | /2017/ |
2017-old |
2017 | 847,348 | /2017-old/ |
2017-old2 |
2017 | 679,764 | /2017-old2/ |
The three 2017 datasets are successive publications of the same exam and they disagree; all three are kept so the differences stay inspectable.
Layout
index.html + src/ the frontend — one app serving all four datasets and the hub
datasets.js the four dataset ids and their per-dataset content
router.js pathname → dataset
data/<id>/ raw Excel files, one directory per dataset
parser/ the Rust parser
src/schema.rs canonical 22-column table: DDL, INSERT, subject regexes
configs/<id>.toml per-dataset parse rules only, no SQL
scripts/ database build, crawler, parity verification
scripts/ site assembly
docs/ architecture, data pipeline, deployment
The dataset id is one identifier end to end:
data/2017-old/ → parser/configs/2017-old.toml → db/2017-old.db.gz → /thptqg/2017-old/
Build
npm ci
npm run build:rust # compile the parser
npm run build:db # build + gzip all four databases (add an id for just one)
npm run build:site # one Vite build, then assemble into _site/
npx serve _site
Pushing to main runs the same steps in
.github/workflows/deploy-pages.yml and publishes to GitHub Pages.
Adding a dataset
- Put the Excel files in
data/<id>/ - Add
parser/configs/<id>.toml— sheet mode, column indices, validation guards. No SQL; the schema is canonical. - Add an entry to
DATASETSinsrc/datasets.js
Everything else follows: the build script, the site assembly and the router all read that one list, and the UI adapts to whichever columns the dataset fills.
Docs
See docs/ — overview,
architecture,
data pipeline,
deployment.