fix(parser): read the two 2016 layouts that were column-shifted

Four of the 119 files in data/2016 publish one score column per subject
instead of a DIEM_THI sentence, and none of them was being read correctly.

The ĐH Công nghiệp Thực phẩm file puts a three-row ministry title block
above its header, so no header was recognised and the positional fallback
shifted every column by one: the serial number became so_bao_danh, the
exam number became ho_ten, the name became ngay_sinh, and the national ID
became the score cell. All 7,833 rows were unusable. The three ĐH Cần Thơ
files name an SBD column but no DIEM_THI, so they fell to the same
fallback: surname into ngay_sinh, given name into ten_cum_thi, birth date
into the score cell, and 12,152 candidates with no scores at all.

Both are now read by FormatSubjectColumns, which resolves identity and one
column per subject from the header. The header is searched for in the
first five rows, so a title block no longer hides it.

The Cần Thơ score columns are numbered rather than named. They follow the
order the exam was sat — each morning an essay paper, each afternoon a
multiple-choice one — which is what identifies them: columns 1/3/5/7
quantise to 0.25 and 2/4/6/8 do not, and each column's mean lands within
0.5 of the same subject's mean across the rest of the dataset. The
foreign language is filed under the subject its N1..N6 code names.

Gender now accepts the 0/1 encoding those files use: of the rows marked
1, 53% carry "Thị" in the name against 1% of those marked 0. Birth dates
in the compact ddmmyy form are expanded so the column holds one format.

A score of 0 is stored rather than dropped, recovering 302 real scores
that a JavaScript falsy check had been turning into NULL.

Row count falls by one, to 877,460: the removed row is the title line
"ĐƠN VỊ: / TRƯỜNG ĐẠI HỌC CÔNG NGHIỆP THỰC PHẨM TP. HỒ CHÍ MINH", which
had been stored as a student. The dataset has no duplicate exam numbers;
the three rows previously described as collapsing were that same file's
title and header lines being counted and then rejected.

Also drops behaviour that existed only to match the parser this one
replaced: the inert "SINH " header token, the untrimmed diem_thi cell, an
unreachable blank-row branch, and a cross-check test against a database
that can no longer exist. None of them changes output.

Verified by rebuilding both datasets: 877,460 and 861,068 rows, both
artifacts through the assembler's row and size guards, and the reader
fidelity suite unchanged across all 182 files.
This commit is contained in:
tiennm99 committed 2026-08-14 10:38:59 +07:00
1 parent 26d0ba81ce
commit 219c7c6a69
15 files changed
+535 -237

No files matched your search

+2 -11
View File
@@ -68,9 +68,7 @@ type ParsedRow struct {
//
// The distinction is load-bearing for the printed counters: the two non-blank
// reasons count as source rows while SkipBlankRow does not. That split lives in
// the CALLER, not here — the build loop drops blank rows before the source-row
// counter and, at the later switch, lets SkipBlankRow fall through to transform
// and insert. Both call sites have to stay as they are.
// the CALLER — the build loop drops blank rows before the source-row counter.
type SkipReason int
const (
@@ -188,14 +186,7 @@ func TransformRow(raw []reader.Cell, cfg *config.DatasetConfig) (*ParsedRow, err
hoTen := get(cols.HoTen)
ngaySinh := get(cols.NgaySinh)
soBaoDanh := get(cols.SoBaoDanh)
// diem_thi is read WITHOUT trimming, unlike the three fields above. Harmless
// because the score patterns are unanchored, but it is the shipped behaviour
// — do not "tidy" it.
diemThi := ""
if cols.DiemThi >= 0 && cols.DiemThi < len(raw) {
diemThi = raw[cols.DiemThi].Str
}
diemThi := get(cols.DiemThi)
var ngaySinhOpt *string
if ngaySinh != "" {