mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-10-17 18:14:44 +00:00
fix(parser): read the two 2016 layouts that were column-shifted
Four of the 119 files in data/2016 publish one score column per subject instead of a DIEM_THI sentence, and none of them was being read correctly. The ĐH Công nghiệp Thực phẩm file puts a three-row ministry title block above its header, so no header was recognised and the positional fallback shifted every column by one: the serial number became so_bao_danh, the exam number became ho_ten, the name became ngay_sinh, and the national ID became the score cell. All 7,833 rows were unusable. The three ĐH Cần Thơ files name an SBD column but no DIEM_THI, so they fell to the same fallback: surname into ngay_sinh, given name into ten_cum_thi, birth date into the score cell, and 12,152 candidates with no scores at all. Both are now read by FormatSubjectColumns, which resolves identity and one column per subject from the header. The header is searched for in the first five rows, so a title block no longer hides it. The Cần Thơ score columns are numbered rather than named. They follow the order the exam was sat — each morning an essay paper, each afternoon a multiple-choice one — which is what identifies them: columns 1/3/5/7 quantise to 0.25 and 2/4/6/8 do not, and each column's mean lands within 0.5 of the same subject's mean across the rest of the dataset. The foreign language is filed under the subject its N1..N6 code names. Gender now accepts the 0/1 encoding those files use: of the rows marked 1, 53% carry "Thị" in the name against 1% of those marked 0. Birth dates in the compact ddmmyy form are expanded so the column holds one format. A score of 0 is stored rather than dropped, recovering 302 real scores that a JavaScript falsy check had been turning into NULL. Row count falls by one, to 877,460: the removed row is the title line "ĐƠN VỊ: / TRƯỜNG ĐẠI HỌC CÔNG NGHIỆP THỰC PHẨM TP. HỒ CHÍ MINH", which had been stored as a student. The dataset has no duplicate exam numbers; the three rows previously described as collapsing were that same file's title and header lines being counted and then rejected. Also drops behaviour that existed only to match the parser this one replaced: the inert "SINH " header token, the untrimmed diem_thi cell, an unreachable blank-row branch, and a cross-check test against a database that can no longer exist. None of them changes output. Verified by rebuilding both datasets: 877,460 and 861,068 rows, both artifacts through the assembler's row and size guards, and the reader fidelity suite unchanged across all 182 files.
This commit is contained in:
1 parent
26d0ba81ce
commit
219c7c6a69
15 files changed
+535
-237
No files matched your search
@@ -68,9 +68,7 @@ type ParsedRow struct {
|
||||
//
|
||||
// The distinction is load-bearing for the printed counters: the two non-blank
|
||||
// reasons count as source rows while SkipBlankRow does not. That split lives in
|
||||
// the CALLER, not here — the build loop drops blank rows before the source-row
|
||||
// counter and, at the later switch, lets SkipBlankRow fall through to transform
|
||||
// and insert. Both call sites have to stay as they are.
|
||||
// the CALLER — the build loop drops blank rows before the source-row counter.
|
||||
type SkipReason int
|
||||
|
||||
const (
|
||||
@@ -188,14 +186,7 @@ func TransformRow(raw []reader.Cell, cfg *config.DatasetConfig) (*ParsedRow, err
|
||||
hoTen := get(cols.HoTen)
|
||||
ngaySinh := get(cols.NgaySinh)
|
||||
soBaoDanh := get(cols.SoBaoDanh)
|
||||
|
||||
// diem_thi is read WITHOUT trimming, unlike the three fields above. Harmless
|
||||
// because the score patterns are unanchored, but it is the shipped behaviour
|
||||
// — do not "tidy" it.
|
||||
diemThi := ""
|
||||
if cols.DiemThi >= 0 && cols.DiemThi < len(raw) {
|
||||
diemThi = raw[cols.DiemThi].Str
|
||||
}
|
||||
diemThi := get(cols.DiemThi)
|
||||
|
||||
var ngaySinhOpt *string
|
||||
if ngaySinh != "" {
|
||||
|
||||
Reference in new issue
Block a user