mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-10-11 03:13:48 +00:00
tiennm99
cd4b07f9cb
The 2016 and 2017 parsers were the same crate with divergent SQL. DDL, INSERT and the subject regex table lived in each dataset's TOML config, so four copies had to be kept in step by hand — which is how the two schemas drifted apart. Move all of it into src/schema.rs as a single 22-column definition: 6 identity columns (adding ten_cum_thi and gioi_tinh, previously 2016-only) and 16 subject columns (the union of both exam years). Columns a dataset carries no data for bind NULL, costing ~1 byte per row. This collapses writer.rs to one insert path: insert_row_2016, SCORE_FIELDS_2016, SCORE_FIELDS_2017 and the SCORE_FIELDS alias all go away. Configs shrink to the parse rules that genuinely vary per dataset — sheet mode, column indices, SBD validation, header tokens, blank-row stripping — and carry no SQL at all. Config parsing now denies unknown fields, so a leftover [schema] block fails loudly instead of looking effective while schema.rs drives the build. idx_ten_cum_thi is partial, so it holds zero entries on the three 2017 datasets where the column is always NULL. Adds db-stats.js and verify-parity.js to prove no data moved: row counts, per-column non-NULL counts and a deterministic student sample are compared against databases built from the previous code. Verified across all four datasets — row counts, every pre-existing column count, and all sampled students are identical. Database size grows 1.4-2.2%.