tiennm99
3fcfe4862d
The docs still described two standalone projects with separate frontends,
separate parsers and three Vite variants. Rewritten around what the repo now
is, merging both projects' copies rather than keeping one and discarding the
other — deployment-guide.md and system-architecture.md existed in both and
documented different pipelines.
project-overview.md goal, scope, constraints, the four datasets, history
system-architecture.md data flow, canonical schema, routing, how one
frontend serves both exam years without branching
data-pipeline.md per-dataset Excel formats, the three 2016 layouts,
overflow-sheet gotcha, expected row counts
deployment-guide.md the single-build workflow, adding a dataset,
why no uncompressed database can ship
Records the release gate in plans/reports/parser-parity-result.md: row counts,
all 18 pre-existing per-column non-NULL counts and the deterministic student
samples are identical across all four datasets, against databases decompressed
from the exact bytes the pipeline publishes. The 1,691 recovered
foreign-language scores are documented with the evidence they are real.
Also drops a machine-specific absolute path from a comment in
format_detect_2016.rs. The build-database.js citations there are kept: that
file no longer exists in this repo, but the references explain why several
parsing rules look arbitrary.
2.6 KiB
Parser parity result
Comparison of the databases the unified pipeline ships against a baseline built
from the pre-refactor code. Run on 2026-08-13 against the final tree
(npm run build:rust && npm run build:db), gate exit code 0.
Inputs:
plans/reports/parser-parity-baseline.json— built with the two separate crates, before any changeplans/reports/parser-parity-current.json— decompressed from.build/public/db/*.db.gz, i.e. the exact bytes published
Result
| Dataset | Rows | Columns | Size |
|---|---|---|---|
| 2016 | 877,461 | 18 → 22 | +1.4% |
| 2017 | 861,068 | 18 → 22 | +2.2% |
| 2017-old | 847,348 | 18 → 22 | +2.2% |
| 2017-old2 | 679,764 | 18 → 22 | +2.2% |
Unchanged, for every dataset:
- row count
- non-NULL count for all 18 pre-existing columns
- every field of a deterministic student sample (SBDs ending
0000)
Approved recoveries
Unifying the subject regexes recovered 1,691 foreign-language scores that the old per-year configs discarded. The 2016 config listed 12 subject patterns and the 2017 configs 14; neither list was complete, so candidates who sat German, Japanese or Russian ended up with no foreign-language score at all.
| Dataset | Recovered |
|---|---|
| 2016 | 182 × tieng_nga |
| 2017 | 93 × tieng_duc, 512 × tieng_nhat |
| 2017-old | 85 × tieng_duc, 484 × tieng_nhat |
| 2017-old2 | 22 × tieng_duc, 313 × tieng_nhat |
Confirmed real rather than spurious matches:
- Every student in all four datasets holds zero or exactly one foreign language — never two — so nothing is duplicated.
- Each affected student had all language columns NULL beforehand. SBD
01003198went from all-NULL totieng_duc = 8. - Counts track dataset size across the three 2017 generations (93/85/22 German, 512/484/313 Japanese).
- Values are ordinary exam scores in the 0–10 range.
These exact counts are pinned as APPROVED_RECOVERY in
parser/scripts/verify-parity.js. Any other newly-populated column, or drift
in these numbers, still fails the gate.
Columns that must stay empty
Verified 0 non-NULL:
- 2016:
khtn,khxh,gdcd - 2017, 2017-old, 2017-old2:
ten_cum_thi,gioi_tinh
Reproducing
npm run build:rust && npm run build:db
node parser/scripts/db-stats.js \
2016=<db> 2017=<db> 2017-old=<db> 2017-old2=<db> > current.json
node parser/scripts/verify-parity.js \
plans/reports/parser-parity-baseline.json current.json
The baseline cannot be regenerated — it required the two pre-refactor crates, which no longer exist. It is committed for that reason.
Unresolved questions
None.