Merge pull request #8 from tiennm99/fix/2016-source-layouts

fix(parser): read the two 2016 layouts that were column-shifted

CLAUDE.md

@@ -34,16 +34,20 @@ hashes every real input file. That is the point of it; do not skip it.
 ## Things that look wrong but are not
 
 - **`data/2016/` filenames are content hashes and must stay verbatim.** The
-  parser sorts inputs bytewise and inserts last-wins, so filenames decide which
-  row survives a duplicate exam number — 877,464 source rows collapse to
-  877,461. Renaming them also breaks `parser/testdata/reader-fidelity-hashes.tsv`,
-  which is keyed by full path.
+  parser sorts inputs bytewise and inserts last-wins, so filenames would decide
+  which row survives a duplicate exam number. Renaming them also breaks
+  `parser/testdata/reader-fidelity-hashes.tsv`, which is keyed by full path.
 - **`parser/testdata/reader-fidelity-hashes.tsv` is frozen and cannot be
   regenerated.** A mismatch is a reader bug until proven otherwise, never a cue
   to refresh the file. `parser/cmd/dumpcells` narrows a failure to the cell.
 - **`ToAscii` filters the literal range U+0300..U+036F, not `unicode.Mn`.** It
   must match `toAscii` in `web/src/App.jsx`, or accent-insensitive search
   silently misses rows.
+- **The 2016 files use four different layouts, and detection is per sheet.**
+  Two of them publish scores in one column per subject instead of a `DIEM_THI`
+  sentence, and one puts a three-row ministry title block above its header.
+  `parser/internal/ingest/detect2016.go` holds the header tables; they are
+  observations about 119 specific files, not a rule to generalise.
 - **`base` in `web/vite.config.js` is absolute.** One emitted `index.html` is
   copied to every dataset path, so every URL is a real static file and no SPA
   404-fallback is needed — a fallback would break the `?q=` deep links.

README.md

@@ -9,7 +9,7 @@ Live at **[tiennm99.github.io/thptqg](https://tiennm99.github.io/thptqg/)**.
 
 | Dataset | Exam | Candidates | Site |
 | --- | --- | --- | --- |
-| `2016` | 2016 | 877,461 | [/2016/](https://tiennm99.github.io/thptqg/2016/) |
+| `2016` | 2016 | 877,460 | [/2016/](https://tiennm99.github.io/thptqg/2016/) |
 | `2017` | 2017 | 861,068 | [/2017/](https://tiennm99.github.io/thptqg/2017/) |
 
 Two earlier 2017 publications (`2017-old`, `2017-old2`) were kept for a while

assembler/go.mod

@@ -1,6 +1,6 @@
 module github.com/tiennm99/thptqg/assembler
 
-go 1.26.5
+go 1.26.6
 
 require modernc.org/sqlite v1.56.0
 

crawler/go.mod

@@ -1,6 +1,6 @@
 module github.com/tiennm99/thptqg/crawler
 
-go 1.26.5
+go 1.26.6
 
 require (
 	golang.org/x/net v0.58.0

datasets.json

@@ -18,8 +18,8 @@
   "datasets": [
     {
       "id": "2016",
-      "expectedRows": 877461,
-      "dbSizeMb": 44
+      "expectedRows": 877460,
+      "dbSizeMb": 45
     },
     {
       "id": "2017",

docs/data-pipeline.md

@@ -92,17 +92,29 @@ server-assigned names verbatim, since it did not choose them.
 | 2 | SOBAODANH | 8-digit string, first 2 digits = province |
 | 3 | DIEM_THI | concatenated per-subject scores, e.g. `"Toán: 6.80 Ngữ văn: 5.25 …"` |
 
-2016 has **three** layouts across its 119 files, chosen per file at runtime via
+2016 has **four** layouts across its 119 files, chosen per sheet at runtime via
 `format_detection: thptqg2016` in its config:
 
 | Format | Detected by | Notes |
 | --- | --- | --- |
 | `separate-scores` | `row[0] == "SBD"` and `row[2] == "TOAN"` | one column per subject; scores read directly, no regex |
 | `mapped` | header contains `SOBAODANH`/`SBD` plus `DIEM_THI` | column indices derived from the header |
+| `subject-columns` | header names three or more subject columns | university-cluster files; identity and scores both resolved by header name |
 | `default` | no recognised header | positional 6-column layout |
 
-The `mapped` and `default` layouts also carry `TEN_CUMTHI` and `GIOI_TINH`,
-which is why only 2016 populates those columns.
+The header may sit below a title block, so the first five rows are searched for
+it; rows above it are not data.
+
+`subject-columns` covers two spellings. The ĐH Công nghiệp Thực phẩm file names
+its columns `TO VA LI HO SI SU DI NN` with the language code in `Môn NN`; the
+three ĐH Cần Thơ files publish `cdiem1..cdiem8`, numbered in the order the 2016
+exam was sat — Toán, Ngoại ngữ, Ngữ văn, Vật lí, Địa lí, Hóa học, Lịch sử, Sinh
+học — with the language code in `ngoaingu`. The language score is filed under
+the subject its `N1`..`N6` code names.
+
+The `mapped`, `default` and `subject-columns` layouts also carry `GIOI_TINH`
+(and `mapped`/`default` `TEN_CUMTHI`), which is why only 2016 populates those
+columns.
 
 ## Score text parsing
 
@@ -171,7 +183,7 @@ silently drops 13,720 students** (Hanoi +7,275, HCM +6,445). That is what
 
 | id | Source rows | Skipped | DB rows |
 | --- | --- | --- | --- |
-| `2016` | 877,464 | 3 duplicate SBDs collapsed | **877,461** |
+| `2016` | 877,460 | 0 | **877,460** |
 | `2017` | 861,068 | 0 | **861,068** |
 
 ## Verifying a rebuild

docs/project-overview.md

@@ -34,7 +34,7 @@ running entirely in the browser and hosted for free on GitHub Pages. Covers the
 
 | id | Exam | Candidates | Notes |
 | --- | --- | --- | --- |
-| `2016` | 2016 | 877,461 | 119 files, three column layouts |
+| `2016` | 2016 | 877,460 | 119 files, four column layouts |
 | `2017` | 2017 | 861,068 | current generation of three publications |
 
 Two further 2017 datasets (`2017-old`, `2017-old2`) were kept alongside these

docs/system-architecture.md

@@ -51,7 +51,7 @@ that does not exist.
 
 | id | Exam | Rows | Source |
 | --- | --- | --- | --- |
-| `2016` | 2016 | 877,461 | `dtnt.bacninh.edu.vn` |
+| `2016` | 2016 | 877,460 | `dtnt.bacninh.edu.vn` |
 | `2017` | 2017 | 861,068 | `baotintuc.vn` |
 
 Full source URLs are in [data-pipeline](./data-pipeline.md#sources); the web
@@ -139,9 +139,9 @@ and had drifted apart on exactly this rule).
 | 9 digits with leading zero | `017006021` | 2016 |
 | 2-4 letters then digits | `BAL000001` | 2016 — exam cluster code |
 
-The letter-prefixed form is the majority case for 2016: 616,593 of 877,461
-candidates (70.3%). Letter prefixes are upper-cased before lookup, so
-`bal000001` resolves.
+The letter-prefixed form is the majority case for 2016: 624,424 of 877,460
+candidates (71.2%); the remaining 253,036 are all 9-digit. Letter prefixes are
+upper-cased before lookup, so `bal000001` resolves.
 
 ## Score tiers
 

parser/README.md

@@ -26,7 +26,7 @@ go -C assembler run ./cmd/assemble db
 | path | role |
 |---|---|
 | `internal/reader` | spreadsheet reading; the only place that knows about file formats |
-| `internal/ingest` | dataset policy — sheet selection, header skipping, blank rows, the build loop, and the 2016 per-sheet format detection |
+| `internal/ingest` | dataset policy — sheet selection, header skipping, blank rows, the build loop, and the 2016 per-sheet layout detection |
 | `internal/transform` | `ToAscii`, score-regex parsing, row validation |
 | `internal/schema` | the canonical 22-column table: DDL, INSERT, subject regexes |
 | `internal/config` | per-dataset YAML parse rules |
@@ -37,40 +37,35 @@ The reader deliberately knows nothing about datasets: it reports every sheet and
 every row verbatim. All policy lives in `ingest`. That split is what made the
 reader independently verifiable against a hash oracle.
 
-## Provenance
+## Behaviour worth knowing
 
-This is a port of a Rust crate that occupied this same path until the Go
-implementation reached full parity, when it was built alongside as `go-parser/`
-and moved back here once the Rust was removed. Source comments cite the original
-by file and line (`parser/src/transform.rs:56` and similar) — those refer to the
-Rust tree and resolve at the tag **`pre-go-parser-removal`**, the last commit
-containing it.
-
-The port was gated on a field-by-field comparison of both implementations across
-the four datasets that existed then — 3,265,641 rows with identical full-table
-SHA-256, identical per-column non-NULL counts, identical schema metadata and
-identical build stdout. `assemble verify` is that comparator today and
-still runs against any two sets of databases.
-
-Behaviour was matched bug-for-bug, deliberately. Several quirks look like
-defects and are load-bearing for the published data:
-
-- a parsed score of `0` becomes NULL in the 2016 separate-scores layout,
-  replicating a JavaScript falsy check;
 - `ToAscii` strips combining marks in the literal range U+0300–U+036F rather
-  than by Unicode category, which is narrower;
-- gender is a two-value allowlist, and anything else becomes NULL;
-- `diem_thi` is read untrimmed while the other three fields are trimmed;
-- the `"SINH "` header token carries a trailing space.
+  than by Unicode category. That covers every Vietnamese diacritic and must
+  stay identical to `toAscii` in `web/src/App.jsx`, or accent-insensitive
+  search misses rows.
+- Gender is normalised to `Nam`/`Nữ`; the Cần Thơ files write `0`/`1` instead
+  and are translated. Anything else becomes NULL.
+- A score of `0` is a real score — the candidate sat the paper and scored
+  nothing — and is stored, not dropped.
+- Birth dates are stored as `dd/mm/yyyy`. The Cần Thơ files' compact `ddmmyy`
+  is expanded; the century is always 19xx, since a 2016 candidate born later
+  would have sat the exam under age.
 
-Each has a test naming it, so none can be tidied away by accident.
+This code began as a port of a Rust crate that occupied the same path, and was
+gated on a field-by-field comparison against it. That comparison is over:
+correctness against the source spreadsheets decides behaviour now, not
+agreement with the old implementation. `assemble verify` is the comparator
+that gated it, and still compares any two sets of built databases.
 
 ## Verification
 
 `testdata/reader-fidelity-hashes.tsv` holds a SHA-256 per input file over a
-canonical dump of every cell of every sheet. It is **frozen**: it was produced
-by the Rust reader, which no longer exists, so it cannot be regenerated. It
-still fails if any single cell of any input file reads differently.
+canonical dump of every cell of every sheet. It is **frozen**: the tool that
+produced it no longer exists, so it cannot be regenerated. It still fails if
+any single cell of any input file reads differently, which is what makes it
+useful — cell rendering and sheet geometry are settled, and a change there is
+a regression until proven otherwise. Dataset policy lives in `ingest`, so
+fixing a layout never touches it.
 
 A mismatch names the file but not the cell. `cmd/dumpcells` prints the stream
 the hash is taken over, so two runs can be diffed:

parser/configs/2016.yml

@@ -1,10 +1,11 @@
 # thptqg2016 data/ — 4 .xls + 115 .xlsx mixed files.
 #
-# Three column layouts exist across the 119 files; the binary selects the
-# right one per-file at runtime via format_detection: thptqg2016:
+# Four column layouts exist across the 119 files; the binary selects the
+# right one per-sheet at runtime via format_detection: thptqg2016:
 #
 #   separate-scores  header SBD(0)/HOTEN(1)/TOAN(2)...  — dhhanghai files
 #   mapped           header SOBAODANH|SBD + DIEM_THI     — most provinces
+#   subject-columns  header names >=3 subject columns    — dhcantho, dhcnthucpham
 #   default          no header; positional 6-col layout  — remaining files
 #
 # This is the only dataset that populates ten_cum_thi and gioi_tinh, and the

parser/go.mod

@@ -1,6 +1,6 @@
 module github.com/tiennm99/thptqg/parser
 
-go 1.26.5
+go 1.26.6
 
 require (
 	github.com/pbnjay/grate v0.0.0-20231006022435-3f8e65d74a14

parser/internal/ingest/detect2016.go

@@ -14,14 +14,11 @@ import (
 )
 
 // The 2016 dataset's 119 files were produced by inconsistent tooling and use
-// three different column layouts, chosen per sheet at runtime. The literals
+// four different column layouts, chosen per sheet at runtime. The literals
 // below are institutional knowledge about those files, with no rule to derive
 // them from — extend them, but do not tidy them into something more regular.
 
 // KnownHeaders are the upper-cased first-cell values that identify a header row.
-//
-// NOTE "SINH " carries a TRAILING SPACE. Trimming it would change which rows
-// are recognised as headers.
 var KnownHeaders = []string{
 	"SOBAODANH",
 	"SBD",
@@ -37,7 +34,7 @@ var KnownHeaders = []string{
 	"VAN",
 	"LY",
 	"HOA",
-	"SINH ",
+	"SINH",
 	"SU",
 	"DIA",
 }
@@ -61,7 +58,7 @@ func IsHeaderRow2016(row []reader.Cell) bool {
 	return isKnownHeader(strings.ToUpper(strings.TrimSpace(row[0].Str)))
 }
 
-// FormatKind is one of the three 2016 layouts.
+// FormatKind is one of the 2016 layouts.
 type FormatKind int
 
 const (
@@ -73,9 +70,13 @@ const (
 	// FormatDefault is the positional 6-column layout used when no header is
 	// recognised. It is FormatMapped with fixed indices, not a separate path.
 	FormatDefault
+	// FormatSubjectColumns resolves BOTH identity and one score column per
+	// subject from the header by name — the university-cluster files, which
+	// publish scores in columns instead of a DIEM_THI sentence.
+	FormatSubjectColumns
 )
 
-// Format is a detected layout plus, for the mapped case, its column indices.
+// Format is a detected layout plus its resolved column indices.
 type Format struct {
 	Kind      FormatKind
 	Sbd       int
@@ -84,6 +85,14 @@ type Format struct {
 	TenCumThi *int
 	GioiTinh  *int
 	DiemThi   int
+
+	// Subjects maps a schema score field to its column, for
+	// FormatSubjectColumns only.
+	Subjects map[string]int
+	// LangScore is the foreign-language score column; the subject it counts
+	// towards is named by the code in LangCode.
+	LangScore *int
+	LangCode  *int
 }
 
 // defaultFormat is FormatMapped with the fixed positional indices.
@@ -95,6 +104,90 @@ func defaultFormat() Format {
 	}
 }
 
+// identity holds the identity columns resolved from a header row. A nil field
+// means the header does not name that column.
+type identity struct {
+	Sbd, HoTen, NgaySinh, TenCumThi, GioiTinh, DiemThi *int
+}
+
+// resolveIdentity maps header names to identity columns, order-independent.
+// Each column has several spellings across the corpus, including the accented
+// and unseparated ones the university files use.
+func resolveIdentity(cols []string) identity {
+	var id identity
+	for i := range cols {
+		idx := i
+		switch cols[i] {
+		case "SOBAODANH", "SBD":
+			id.Sbd = &idx
+		case "HO_TEN", "HOTEN", "HỌ TÊN":
+			id.HoTen = &idx
+		case "NGAY_SINH", "NGAYSINH", "NGÀY SINH":
+			id.NgaySinh = &idx
+		case "TEN_CUMTHI":
+			id.TenCumThi = &idx
+		case "GIOI_TINH", "GIỚI TÍNH", "PHAI":
+			id.GioiTinh = &idx
+		case "DIEM_THI":
+			id.DiemThi = &idx
+		}
+	}
+	return id
+}
+
+// subjectColumnFamily is one university-cluster spelling of the per-subject
+// score columns. The two families are told apart by which names resolve.
+type subjectColumnFamily struct {
+	scores    map[string]string // header name -> schema score field
+	langScore string            // header of the foreign-language score column
+	langCode  string            // header of the N1..N6 language code column
+}
+
+// subjectColumnFamilies are tried in order. The Cần Thơ family goes first
+// because its CDIEM<n> names are unambiguous, while the two-letter family
+// shares "HO" with that layout's surname column.
+var subjectColumnFamilies = []subjectColumnFamily{
+	{scores: canThoScoreHeaders, langScore: "CDIEM2", langCode: "NGOAINGU"},
+	{scores: subjectCodeHeaders, langScore: "NN", langCode: "MÔN NN"},
+}
+
+// canThoScoreHeaders maps the Cần Thơ cluster's published-score columns to
+// schema fields. The columns are numbered in the order the 2016 exam was sat —
+// each morning an essay paper (quarter-point scores), each afternoon a
+// multiple-choice one — which is what identifies them: CDIEM1/3/5/7 quantise to
+// 0.25 and CDIEM2/4/6/8 do not, and the per-column means match the published
+// national averages for those subjects.
+var canThoScoreHeaders = map[string]string{
+	"CDIEM1": "toan",
+	"CDIEM3": "ngu_van",
+	"CDIEM4": "vat_ly",
+	"CDIEM5": "dia_ly",
+	"CDIEM6": "hoa_hoc",
+	"CDIEM7": "lich_su",
+	"CDIEM8": "sinh_hoc",
+}
+
+// subjectCodeHeaders maps the two-letter subject columns to schema fields.
+var subjectCodeHeaders = map[string]string{
+	"TO": "toan",
+	"VA": "ngu_van",
+	"LI": "vat_ly",
+	"HO": "hoa_hoc",
+	"SI": "sinh_hoc",
+	"SU": "lich_su",
+	"DI": "dia_ly",
+}
+
+// languageFields maps the ministry's foreign-language codes to schema fields.
+var languageFields = map[string]string{
+	"N1": "tieng_anh",
+	"N2": "tieng_nga",
+	"N3": "tieng_phap",
+	"N4": "tieng_trung",
+	"N5": "tieng_duc",
+	"N6": "tieng_nhat",
+}
+
 // DetectFormat inspects a header row and decides which layout applies.
 func DetectFormat(headerRow []reader.Cell) Format {
 	cols := make([]string, len(headerRow))
@@ -107,54 +200,85 @@ func DetectFormat(headerRow []reader.Cell) Format {
 		return Format{Kind: FormatSeparateScores}
 	}
 
-	// Format 2: resolve indices by header name, order-independent.
-	var sbdIdx, hoTenIdx, ngaySinhIdx, tenCumThiIdx, gioiTinhIdx, diemThiIdx *int
-	for i := range cols {
-		idx := i
-		switch cols[i] {
-		case "SOBAODANH", "SBD":
-			sbdIdx = &idx
-		case "HO_TEN", "HOTEN", "HỌ TÊN":
-			hoTenIdx = &idx
-		case "NGAY_SINH":
-			ngaySinhIdx = &idx
-		case "TEN_CUMTHI":
-			tenCumThiIdx = &idx
-		case "GIOI_TINH":
-			gioiTinhIdx = &idx
-		case "DIEM_THI":
-			diemThiIdx = &idx
+	id := resolveIdentity(cols)
+
+	// Format 2: a free-text DIEM_THI cell alongside an SBD.
+	if id.Sbd != nil && id.DiemThi != nil {
+		hoTen := 1 // fallback: col 1, present in all known files
+		if id.HoTen != nil {
+			hoTen = *id.HoTen
+		}
+		return Format{
+			Kind: FormatMapped, Sbd: *id.Sbd, HoTen: hoTen,
+			NgaySinh: id.NgaySinh, TenCumThi: id.TenCumThi, GioiTinh: id.GioiTinh,
+			DiemThi: *id.DiemThi,
 		}
 	}
 
-	if sbdIdx != nil && diemThiIdx != nil {
-		hoTen := 1 // fallback: col 1, present in all known files
-		if hoTenIdx != nil {
-			hoTen = *hoTenIdx
-		}
-		return Format{
-			Kind: FormatMapped, Sbd: *sbdIdx, HoTen: hoTen,
-			NgaySinh: ngaySinhIdx, TenCumThi: tenCumThiIdx, GioiTinh: gioiTinhIdx,
-			DiemThi: *diemThiIdx,
-		}
+	// Format 3: one column per subject.
+	if f, ok := subjectColumnsFormat(cols, id); ok {
+		return f
 	}
 
 	// Unrecognised header, or none at all.
 	return defaultFormat()
 }
 
-// parseFloatCell parses a per-subject score cell.
+// subjectColumnsFormat resolves a per-subject-column layout, if the header
+// names one.
 //
-// A parsed 0.0 becomes "no score", so a genuine zero is indistinguishable from
-// a blank. Not obviously correct, but it is the shipped behaviour and the
-// published data depends on it.
-func parseFloatCell(row []reader.Cell, idx int) (float64, bool) {
+// Three resolved subjects are required: a stray one- or two-letter header in an
+// unrelated file must not turn that file into this layout.
+func subjectColumnsFormat(cols []string, id identity) (Format, bool) {
+	if id.Sbd == nil {
+		return Format{}, false
+	}
+	for _, fam := range subjectColumnFamilies {
+		subjects := make(map[string]int)
+		for i, name := range cols {
+			if field, ok := fam.scores[name]; ok {
+				subjects[field] = i
+			}
+		}
+		if len(subjects) < 3 {
+			continue
+		}
+		hoTen := 1
+		if id.HoTen != nil {
+			hoTen = *id.HoTen
+		}
+		return Format{
+			Kind: FormatSubjectColumns, Sbd: *id.Sbd, HoTen: hoTen,
+			NgaySinh: id.NgaySinh, TenCumThi: id.TenCumThi, GioiTinh: id.GioiTinh,
+			DiemThi:   -1,
+			Subjects:  subjects,
+			LangScore: indexOf(cols, fam.langScore),
+			LangCode:  indexOf(cols, fam.langCode),
+		}, true
+	}
+	return Format{}, false
+}
+
+// indexOf returns the column holding name, or nil when the header lacks it.
+func indexOf(cols []string, name string) *int {
+	for i := range cols {
+		if cols[i] == name {
+			idx := i
+			return &idx
+		}
+	}
+	return nil
+}
+
+// parseScoreCell parses a per-subject score cell. A parsed 0 is a real score:
+// the candidate sat the paper and scored nothing.
+func parseScoreCell(row []reader.Cell, idx int) (float64, bool) {
 	s := cellAt(row, idx)
 	if s == "" {
 		return 0, false
 	}
 	v, err := strconv.ParseFloat(s, 64)
-	if err != nil || v == 0.0 {
+	if err != nil {
 		return 0, false
 	}
 	return v, true
@@ -182,7 +306,7 @@ func processSeparateScoresRow(row []reader.Cell) *transform.ParsedRow {
 		"sinh_hoc": 6, "lich_su": 7, "dia_ly": 8,
 		"tieng_anh": 11, // NGOAINGU total
 	} {
-		if v, ok := parseFloatCell(row, idx); ok {
+		if v, ok := parseScoreCell(row, idx); ok {
 			scores[field] = v
 		}
 	}
@@ -209,47 +333,146 @@ func processMappedRow(row []reader.Cell, f Format) *transform.ParsedRow {
 		return nil
 	}
 
-	optional := func(idx *int) *string {
-		if idx == nil {
-			return nil
-		}
-		if s := cellAt(row, *idx); s != "" {
-			return &s
-		}
+	return &transform.ParsedRow{
+		SoBaoDanh:  sbd,
+		HoTen:      hoTen,
+		HoTenAscii: transform.ToAscii(hoTen),
+		NgaySinh:   birthDate(row, f.NgaySinh),
+		TenCumThi:  optionalCell(row, f.TenCumThi),
+		GioiTinh:   gender(row, f.GioiTinh),
+		Scores:     transform.ParseScores(cellAt(row, f.DiemThi)),
+	}
+}
+
+// processSubjectColumnsRow reads a row whose scores sit in one column per
+// subject, with the foreign language filed under the subject its code names.
+func processSubjectColumnsRow(row []reader.Cell, f Format) *transform.ParsedRow {
+	sbd := cellAt(row, f.Sbd)
+	hoTen := cellAt(row, f.HoTen)
+	if sbd == "" || hoTen == "" {
+		return nil
+	}
+	if isKnownHeader(strings.ToUpper(sbd)) || isKnownHeader(strings.ToUpper(hoTen)) {
 		return nil
 	}
 
-	// Gender is a two-value allowlist, not a general enum: anything other than
-	// exactly "Nam" or "Nữ" becomes nil.
-	var gioiTinh *string
-	if f.GioiTinh != nil {
-		if s := cellAt(row, *f.GioiTinh); s == "Nam" || s == "Nữ" {
-			gioiTinh = &s
+	scores := make(map[string]float64)
+	for field, idx := range f.Subjects {
+		if v, ok := parseScoreCell(row, idx); ok {
+			scores[field] = v
 		}
 	}
-
-	// The free-text score cell is read untrimmed, unlike every other cell here.
-	diemThi := ""
-	if f.DiemThi >= 0 && f.DiemThi < len(row) {
-		diemThi = row[f.DiemThi].Str
+	if f.LangScore != nil {
+		if v, ok := parseScoreCell(row, *f.LangScore); ok {
+			code := "N1" // an unlabelled language paper is English in this corpus
+			if f.LangCode != nil {
+				if c := strings.ToUpper(cellAt(row, *f.LangCode)); c != "" {
+					code = c
+				}
+			}
+			// An unknown code is left out rather than guessed at.
+			if field, ok := languageFields[code]; ok {
+				scores[field] = v
+			}
+		}
 	}
 
 	return &transform.ParsedRow{
 		SoBaoDanh:  sbd,
 		HoTen:      hoTen,
 		HoTenAscii: transform.ToAscii(hoTen),
-		NgaySinh:   optional(f.NgaySinh),
-		TenCumThi:  optional(f.TenCumThi),
-		GioiTinh:   gioiTinh,
-		Scores:     transform.ParseScores(diemThi),
+		NgaySinh:   birthDate(row, f.NgaySinh),
+		TenCumThi:  optionalCell(row, f.TenCumThi),
+		GioiTinh:   gender(row, f.GioiTinh),
+		Scores:     scores,
 	}
 }
 
+// optionalCell returns the trimmed cell at idx, or nil when the column is
+// absent or its cell empty.
+func optionalCell(row []reader.Cell, idx *int) *string {
+	if idx == nil {
+		return nil
+	}
+	if s := cellAt(row, *idx); s != "" {
+		return &s
+	}
+	return nil
+}
+
+// gender normalises the gender cell to "Nam"/"Nữ", or nil when it says neither.
+//
+// The Cần Thơ files encode it numerically instead: of the rows marked 1, 53%
+// carry the female marker "Thị" in the name, against 1% of those marked 0.
+func gender(row []reader.Cell, idx *int) *string {
+	s := optionalCell(row, idx)
+	if s == nil {
+		return nil
+	}
+	switch *s {
+	case "Nam", "Nữ":
+		return s
+	case "0":
+		nam := "Nam"
+		return &nam
+	case "1":
+		nu := "Nữ"
+		return &nu
+	}
+	return nil
+}
+
+// birthDate returns the birth-date cell as dd/mm/yyyy.
+//
+// Most files already write it that way; the Cần Thơ ones use a compact ddmmyy,
+// expanded here so the column holds one format. The century is always 19xx: a
+// 2016 candidate born after 1999 would have sat the exam under age.
+func birthDate(row []reader.Cell, idx *int) *string {
+	s := optionalCell(row, idx)
+	if s == nil || len(*s) != 6 || !allDigits(*s) {
+		return s
+	}
+	out := (*s)[0:2] + "/" + (*s)[2:4] + "/19" + (*s)[4:6]
+	return &out
+}
+
+func allDigits(s string) bool {
+	for i := 0; i < len(s); i++ {
+		if s[i] < '0' || s[i] > '9' {
+			return false
+		}
+	}
+	return true
+}
+
+// headerScanRows is how far into a sheet the header may sit. One file opens
+// with a three-row ministry title block above its header; a sheet with no
+// header at all has data from row 0 and simply finds nothing in that window.
+const headerScanRows = 5
+
+// sheetFormat picks the layout for one sheet and returns the row its data
+// starts on. Rows above the header are a title block and are not data.
+func sheetFormat(rows [][]reader.Cell) (Format, int) {
+	limit := headerScanRows
+	if len(rows) < limit {
+		limit = len(rows)
+	}
+	for i := 0; i < limit; i++ {
+		if IsHeaderRow2016(rows[i]) {
+			return DetectFormat(rows[i]), i + 1
+		}
+	}
+	return defaultFormat(), 0
+}
+
 // ProcessRow2016 dispatches a data row through the detected layout. A nil return
 // means the row is empty or invalid and should be skipped.
 func ProcessRow2016(row []reader.Cell, f Format) *transform.ParsedRow {
-	if f.Kind == FormatSeparateScores {
+	switch f.Kind {
+	case FormatSeparateScores:
 		return processSeparateScoresRow(row)
+	case FormatSubjectColumns:
+		return processSubjectColumnsRow(row, f)
 	}
 	// FormatMapped and FormatDefault share one implementation; Default is just a
 	// fixed index tuple.
@@ -336,12 +559,7 @@ func processFile2016(path string, cfg *config.DatasetConfig, ins *writer.Inserte
 			continue
 		}
 
-		f := defaultFormat()
-		startIdx := 0
-		if IsHeaderRow2016(rows[0]) {
-			f = DetectFormat(rows[0])
-			startIdx = 1
-		}
+		f, startIdx := sheetFormat(rows)
 
 		for _, row := range rows[startIdx:] {
 			// Rows shorter than 2 cells are dropped BEFORE the counter, so they

parser/internal/ingest/detect2016_test.go

@@ -1,6 +1,7 @@
 package ingest
 
 import (
+	"strings"
 	"testing"
 
 	"github.com/tiennm99/thptqg/parser/internal/reader"
@@ -31,20 +32,45 @@ func TestIsHeaderRow2016(t *testing.T) {
 	}
 }
 
-// TestKnownHeadersHasTrailingSpaceToken pins the "SINH " literal. Trimming it
-// would silently change which rows count as headers.
-func TestKnownHeadersHasTrailingSpaceToken(t *testing.T) {
-	var found bool
+// TestKnownHeadersAreTrimmed: the cell is trimmed before comparison, so a token
+// carrying surrounding space could never match anything.
+func TestKnownHeadersAreTrimmed(t *testing.T) {
 	for _, h := range KnownHeaders {
-		if h == "SINH " {
-			found = true
+		if h != strings.TrimSpace(h) {
+			t.Errorf("KnownHeaders token %q has surrounding space and can never match", h)
 		}
 	}
-	if !found {
-		t.Error(`KnownHeaders must contain "SINH " WITH its trailing space`)
+}
+
+// TestSheetFormatSkipsTitleBlock: one file opens with a ministry title block
+// above its header row, and the rows above the header are not data.
+func TestSheetFormatSkipsTitleBlock(t *testing.T) {
+	rows := [][]reader.Cell{
+		cells("BỘ GIÁO DỤC VÀ ĐÀO TẠO", "", ""),
+		cells("ĐƠN VỊ:", "TRƯỜNG ĐẠI HỌC X", ""),
+		cells("DANH SÁCH ĐIỂM THÍ SINH", "", ""),
+		cells("STT", "SBD", "Họ tên", "Ngày sinh", "Giới tính", "CMND", "Tỉnh", "TO", "VA", "LI"),
+		cells("1", "DCT000001", "Nguyễn Văn A", "01/01/1998", "Nam", "123", "46", "8", "7", "6"),
 	}
-	if len(KnownHeaders) != 17 {
-		t.Errorf("KnownHeaders has %d tokens, want 17", len(KnownHeaders))
+	f, start := sheetFormat(rows)
+	if start != 4 {
+		t.Errorf("data starts at row %d, want 4", start)
+	}
+	if f.Kind != FormatSubjectColumns {
+		t.Errorf("Kind = %v, want FormatSubjectColumns", f.Kind)
+	}
+}
+
+// TestSheetFormatHeaderlessSheet: a sheet whose first row is already data keeps
+// the positional layout and starts at row 0.
+func TestSheetFormatHeaderlessSheet(t *testing.T) {
+	rows := [][]reader.Cell{
+		cells("YTB000001", "PHẠM THỊ HỒNG ÁI", "21/08/1998", "Y Dược Thái Bình", "Nữ", "Toán: 7.75"),
+		cells("YTB000002", "TRẦN VĂN B", "22/08/1998", "Y Dược Thái Bình", "Nam", "Toán: 5"),
+	}
+	f, start := sheetFormat(rows)
+	if start != 0 || f.Kind != FormatDefault {
+		t.Errorf("got kind=%v start=%d, want FormatDefault at 0", f.Kind, start)
 	}
 }
 
@@ -133,36 +159,37 @@ func TestProcessSeparateScoresRow(t *testing.T) {
 	}
 }
 
-// TestZeroScoreBecomesNull pins the quirk that a literal 0 is indistinguishable
-// from "no score".
-func TestZeroScoreBecomesNull(t *testing.T) {
+// TestZeroScoreIsKept: a candidate who sat a paper and scored nothing has a
+// score of 0, not a missing one.
+func TestZeroScoreIsKept(t *testing.T) {
 	row := cells("1000", "A", "0", "0.0", "1", "", "", "", "", "", "", "")
 	got := ProcessRow2016(row, Format{Kind: FormatSeparateScores})
 	if got == nil {
 		t.Fatal("row rejected")
 	}
-	if _, ok := got.Scores["toan"]; ok {
-		t.Error(`a "0" score must become NULL, not 0`)
+	for field, want := range map[string]float64{"toan": 0, "ngu_van": 0, "vat_ly": 1} {
+		v, ok := got.Scores[field]
+		if !ok || v != want {
+			t.Errorf("%s = %v (present=%v), want %v", field, v, ok, want)
+		}
 	}
-	if _, ok := got.Scores["ngu_van"]; ok {
-		t.Error(`a "0.0" score must become NULL, not 0`)
-	}
-	if got.Scores["vat_ly"] != 1 {
-		t.Error("a non-zero score must survive")
+	if _, ok := got.Scores["hoa_hoc"]; ok {
+		t.Error("an empty cell must stay absent")
 	}
 }
 
-// TestGenderAllowlist: exactly two accepted values, matched case-sensitively.
-func TestGenderAllowlist(t *testing.T) {
+// TestGenderNormalisation: "Nam"/"Nữ" verbatim, the Cần Thơ files' 1/0 encoding
+// translated, everything else NULL.
+func TestGenderNormalisation(t *testing.T) {
 	f := defaultFormat()
-	for in, want := range map[string]string{"Nam": "Nam", "Nữ": "Nữ"} {
+	for in, want := range map[string]string{"Nam": "Nam", "Nữ": "Nữ", "1": "Nữ", "0": "Nam"} {
 		row := cells("1", "A", "", "", in, "")
 		got := ProcessRow2016(row, f)
 		if got == nil || got.GioiTinh == nil || *got.GioiTinh != want {
-			t.Errorf("gioi_tinh for %q not preserved", in)
+			t.Errorf("gioi_tinh for %q = %v, want %q", in, got.GioiTinh, want)
 		}
 	}
-	for _, in := range []string{"nam", "NAM", "Unknown", "M", "F", "nữ", ""} {
+	for _, in := range []string{"nam", "NAM", "Unknown", "M", "F", "nữ", "", "2"} {
 		row := cells("1", "A", "", "", in, "")
 		got := ProcessRow2016(row, f)
 		if got == nil {
@@ -174,6 +201,131 @@ func TestGenderAllowlist(t *testing.T) {
 	}
 }
 
+// TestBirthDateExpandsCompactForm: ddmmyy becomes dd/mm/19yy so the column
+// holds one format; anything else is passed through.
+func TestBirthDateExpandsCompactForm(t *testing.T) {
+	f := defaultFormat()
+	for in, want := range map[string]string{
+		"140798":     "14/07/1998",
+		"01/01/1998": "01/01/1998",
+		"1998":       "1998",
+	} {
+		got := ProcessRow2016(cells("1", "A", in, "", "", ""), f)
+		if got == nil || got.NgaySinh == nil || *got.NgaySinh != want {
+			t.Errorf("ngay_sinh for %q = %v, want %q", in, got.NgaySinh, want)
+		}
+	}
+}
+
+// --- per-subject score columns ---
+
+// TestDetectFormatSubjectColumnsTwoLetter covers the two-letter subject columns,
+// whose foreign-language score is filed under the subject its code names.
+func TestDetectFormatSubjectColumnsTwoLetter(t *testing.T) {
+	header := cells("STT", "SBD", "Họ tên", "Ngày sinh", "Giới tính", "CMND", "Tỉnh",
+		"TO", "VA", "LI", "HO", "SI", "SU", "DI", "NN", "Môn NN")
+	f := DetectFormat(header)
+	if f.Kind != FormatSubjectColumns {
+		t.Fatalf("Kind = %v, want FormatSubjectColumns", f.Kind)
+	}
+	if f.Sbd != 1 || f.HoTen != 2 {
+		t.Errorf("identity: sbd=%d ho_ten=%d", f.Sbd, f.HoTen)
+	}
+	if f.NgaySinh == nil || *f.NgaySinh != 3 || f.GioiTinh == nil || *f.GioiTinh != 4 {
+		t.Error("accented identity headers not resolved")
+	}
+	if f.Subjects["toan"] != 7 || f.Subjects["dia_ly"] != 13 {
+		t.Errorf("subject columns = %v", f.Subjects)
+	}
+
+	row := cells("1", "DCT000073", "Trần Thị Phước An", "23/11/1998", "Nữ", "291183999", "46",
+		"6.25", "4.5", "5.8", "", "", "", "", "5.38", "N1")
+	got := ProcessRow2016(row, f)
+	if got == nil {
+		t.Fatal("row rejected")
+	}
+	if got.SoBaoDanh != "DCT000073" || got.HoTen != "Trần Thị Phước An" {
+		t.Errorf("identity = %q / %q", got.SoBaoDanh, got.HoTen)
+	}
+	want := map[string]float64{"toan": 6.25, "ngu_van": 4.5, "vat_ly": 5.8, "tieng_anh": 5.38}
+	if len(got.Scores) != len(want) {
+		t.Errorf("scores = %v, want %v", got.Scores, want)
+	}
+	for k, v := range want {
+		if got.Scores[k] != v {
+			t.Errorf("%s = %v, want %v", k, got.Scores[k], v)
+		}
+	}
+}
+
+// TestDetectFormatSubjectColumnsCanTho covers the CDIEM<n> columns, which are
+// numbered in exam-timetable order rather than named.
+func TestDetectFormatSubjectColumnsCanTho(t *testing.T) {
+	header := cells("sbd", "hoten", "ho", "ten", "phai", "ngaysinh", "socmnd",
+		"cdiem1", "cdiem2", "cdiem3", "cdiem4", "cdiem5", "cdiem6", "cdiem7", "cdiem8", "ngoaingu")
+	f := DetectFormat(header)
+	if f.Kind != FormatSubjectColumns {
+		t.Fatalf("Kind = %v, want FormatSubjectColumns", f.Kind)
+	}
+	// "ho" is the surname column here, and must not be read as hoa_hoc.
+	if got, ok := f.Subjects["hoa_hoc"]; ok && got == 2 {
+		t.Error("surname column resolved as a score column")
+	}
+	if f.Subjects["toan"] != 7 || f.Subjects["ngu_van"] != 9 || f.Subjects["sinh_hoc"] != 14 {
+		t.Errorf("subject columns = %v", f.Subjects)
+	}
+	if f.LangScore == nil || *f.LangScore != 8 || f.LangCode == nil || *f.LangCode != 15 {
+		t.Error("language score/code columns not resolved")
+	}
+
+	row := cells("TCT000001", "Dương Diễm ái", "Dương Diễm", "ái", "1", "140798", "362539228",
+		"07.50", "05.60", "06.50", "", "", "08.20", "", "07.60", "N3")
+	got := ProcessRow2016(row, f)
+	if got == nil {
+		t.Fatal("row rejected")
+	}
+	if got.GioiTinh == nil || *got.GioiTinh != "Nữ" {
+		t.Errorf("gioi_tinh = %v, want Nữ", got.GioiTinh)
+	}
+	if got.NgaySinh == nil || *got.NgaySinh != "14/07/1998" {
+		t.Errorf("ngay_sinh = %v", got.NgaySinh)
+	}
+	want := map[string]float64{"toan": 7.5, "tieng_phap": 5.6, "ngu_van": 6.5, "hoa_hoc": 8.2, "sinh_hoc": 7.6}
+	if len(got.Scores) != len(want) {
+		t.Errorf("scores = %v, want %v", got.Scores, want)
+	}
+	for k, v := range want {
+		if got.Scores[k] != v {
+			t.Errorf("%s = %v, want %v", k, got.Scores[k], v)
+		}
+	}
+}
+
+// TestSubjectColumnsUnknownLanguageCode: an unrecognised code drops the score
+// rather than filing it under a guess.
+func TestSubjectColumnsUnknownLanguageCode(t *testing.T) {
+	header := cells("sbd", "hoten", "ho", "ten", "phai", "ngaysinh", "socmnd",
+		"cdiem1", "cdiem2", "cdiem3", "cdiem4", "cdiem5", "cdiem6", "cdiem7", "cdiem8", "ngoaingu")
+	f := DetectFormat(header)
+	row := cells("TCT000002", "B", "", "", "0", "", "", "5", "6", "", "", "", "", "", "", "N9")
+	got := ProcessRow2016(row, f)
+	if got == nil {
+		t.Fatal("row rejected")
+	}
+	if len(got.Scores) != 1 || got.Scores["toan"] != 5 {
+		t.Errorf("scores = %v, want only toan", got.Scores)
+	}
+}
+
+// TestSubjectColumnsNeedsThreeSubjects: a stray two-letter header elsewhere must
+// not turn an unrelated file into this layout.
+func TestSubjectColumnsNeedsThreeSubjects(t *testing.T) {
+	f := DetectFormat(cells("SBD", "HOTEN", "DI", "SU"))
+	if f.Kind == FormatSubjectColumns {
+		t.Error("two subject columns must not be enough to select the layout")
+	}
+}
+
 // TestLeakedHeaderRowSkipped: a header repeated inside the data must not become
 // a student.
 func TestLeakedHeaderRowSkipped(t *testing.T) {

parser/internal/ingest/ingest.go

@@ -174,15 +174,7 @@ func Standard(cfg *config.DatasetConfig, inputDir, outputPath string) error {
 				soBaoDanh = cellAt(row, cols.SoBaoDanh)
 			}
 
-			switch transform.ValidateRow(hoTen, soBaoDanh, &cfg.Validation, stripBlank, allBlank) {
-			case transform.SkipBlankRow:
-				// Falls through to transform and insert. Unreachable here:
-				// BlankRow requires stripBlank && allBlank, which returned
-				// above. Kept so the two call sites with opposite outcomes stay
-				// visibly distinct.
-			case transform.SkipNone:
-				// proceed
-			default:
+			if transform.ValidateRow(hoTen, soBaoDanh, &cfg.Validation, stripBlank, allBlank) != transform.SkipNone {
 				fileSkipped++
 				return
 			}

parser/internal/schema/schema.go

@@ -5,9 +5,9 @@
 //
 // Column provenance:
 //
-//	ten_cum_thi, gioi_tinh, tieng_duc, tieng_nhat  -> 2016 only
-//	khtn, khxh, gdcd, tieng_nga                    -> 2017 datasets only
-//	everything else                                -> both
+//	ten_cum_thi, gioi_tinh  -> 2016 only
+//	khtn, khxh, gdcd        -> 2017 only
+//	everything else         -> both, all six languages included
 //
 // The DDL, the INSERT and the subject regexes belong here and nowhere else. One
 // copy per dataset is what let the 2016 and 2017 schemas drift apart; the
@@ -109,8 +109,8 @@ VALUES
 // the source files — copy them, never retype them.
 //
 // Every pattern runs against every dataset. A subject absent from a given exam
-// year simply never matches and stays NULL: 2016 files contain no "KHTN:" or
-// "Tiếng Nga:" tokens, and 2017 files contain no "Tiếng Đức:" or "Tiếng Nhật:".
+// year simply never matches and stays NULL: 2016 files contain no "KHTN:",
+// "KHXH:" or "GDCD:" tokens, since those combined papers did not exist yet.
 var scorePatternSources = map[string]string{
 	"toan":        `Toán:\s*(\d+(?:\.\d+)?)`,
 	"ngu_van":     `Ngữ văn:\s*(\d+(?:\.\d+)?)`,

parser/internal/transform/ascii_crosscheck_test.go

@@ -1,66 +0,0 @@
-package transform_test
-
-import (
-	"database/sql"
-	"os"
-	"testing"
-
-	_ "github.com/tiennm99/thptqg/parser/internal/sqlitedb"
-	"github.com/tiennm99/thptqg/parser/internal/transform"
-)
-
-// TestToAsciiAgainstRustOutput cross-checks ToAscii against a reference database
-// on real data.
-//
-// Such a database is its own oracle: every row carries ho_ten alongside the
-// ho_ten_ascii derived from it, so the whole table is a name -> expected-slug
-// corpus far broader than the 20 hand-picked unit cases.
-//
-//	GO_PARSER_RUST_DB=/tmp/rust-2016.db go test ./internal/transform/
-//
-// Skips when unset, so the default suite stays hermetic. The reference parser is
-// gone from the tree, so no new oracle database can be produced and this always
-// skips today.
-func TestToAsciiAgainstRustOutput(t *testing.T) {
-	path := os.Getenv("GO_PARSER_RUST_DB")
-	if path == "" {
-		t.Skip("GO_PARSER_RUST_DB not set; skipping cross-check against Rust output")
-	}
-	if _, err := os.Stat(path); err != nil {
-		t.Skipf("GO_PARSER_RUST_DB=%s not readable: %v", path, err)
-	}
-
-	db, err := sql.Open("sqlite", "file:"+path+"?mode=ro")
-	if err != nil {
-		t.Fatalf("open %s: %v", path, err)
-	}
-	defer db.Close()
-
-	rows, err := db.Query("SELECT ho_ten, ho_ten_ascii FROM student")
-	if err != nil {
-		t.Fatalf("query: %v", err)
-	}
-	defer rows.Close()
-
-	var checked, bad int
-	for rows.Next() {
-		var name, rustAscii string
-		if err := rows.Scan(&name, &rustAscii); err != nil {
-			t.Fatalf("scan: %v", err)
-		}
-		checked++
-		if got := transform.ToAscii(name); got != rustAscii {
-			bad++
-			if bad <= 5 {
-				t.Errorf("ToAscii(%q)\n  rust = %q\n  go   = %q", name, rustAscii, got)
-			}
-		}
-	}
-	if err := rows.Err(); err != nil {
-		t.Fatalf("iterate: %v", err)
-	}
-	if checked == 0 {
-		t.Fatal("database contained no rows")
-	}
-	t.Logf("compared %d real names, %d mismatches", checked, bad)
-}

parser/internal/transform/transform.go

@@ -68,9 +68,7 @@ type ParsedRow struct {
 //
 // The distinction is load-bearing for the printed counters: the two non-blank
 // reasons count as source rows while SkipBlankRow does not. That split lives in
-// the CALLER, not here — the build loop drops blank rows before the source-row
-// counter and, at the later switch, lets SkipBlankRow fall through to transform
-// and insert. Both call sites have to stay as they are.
+// the CALLER — the build loop drops blank rows before the source-row counter.
 type SkipReason int
 
 const (
@@ -188,14 +186,7 @@ func TransformRow(raw []reader.Cell, cfg *config.DatasetConfig) (*ParsedRow, err
 	hoTen := get(cols.HoTen)
 	ngaySinh := get(cols.NgaySinh)
 	soBaoDanh := get(cols.SoBaoDanh)
-
-	// diem_thi is read WITHOUT trimming, unlike the three fields above. Harmless
-	// because the score patterns are unanchored, but it is the shipped behaviour
-	// — do not "tidy" it.
-	diemThi := ""
-	if cols.DiemThi >= 0 && cols.DiemThi < len(raw) {
-		diemThi = raw[cols.DiemThi].Str
-	}
+	diemThi := get(cols.DiemThi)
 
 	var ngaySinhOpt *string
 	if ngaySinh != "" {

parser/internal/transform/transform_test.go

@@ -242,10 +242,10 @@ func TestTransformRowShortRowYieldsEmptyFields(t *testing.T) {
 	}
 }
 
-// TestTransformRowDiemThiIsNotTrimmed pins an asymmetry that is easy to
-// "tidy away": ho_ten, ngay_sinh and so_bao_danh are trimmed, but diem_thi is
-// read raw.
-func TestTransformRowDiemThiIsNotTrimmed(t *testing.T) {
+// TestTransformRowTrimsEveryField: every column is read trimmed, diem_thi
+// included. Source cells routinely carry padding — 850k of the 861k 2017 rows
+// have whitespace around their score cell.
+func TestTransformRowTrimsEveryField(t *testing.T) {
 	row := cells("  A  ", "  01/01/2000  ", "  123  ", "   Toán: 5   ")
 	got, err := TransformRow(row, fixedColumnCfg())
 	if err != nil {
@@ -257,7 +257,6 @@ func TestTransformRowDiemThiIsNotTrimmed(t *testing.T) {
 	if got.NgaySinh == nil || *got.NgaySinh != "01/01/2000" {
 		t.Errorf("ngay_sinh = %v, want trimmed", got.NgaySinh)
 	}
-	// Untrimmed diem_thi still parses — the regexes are unanchored.
 	if got.Scores["toan"] != 5 {
 		t.Errorf("scores = %v", got.Scores)
 	}

web/src/datasets.js

@@ -33,7 +33,7 @@ const CONTENT = {
     // ministry — hence the unexpected domain.
     source:
       "https://dtnt.bacninh.edu.vn/tin-tuc/tin-tuc-su-kien/cong-bo-diem-thi-thptqg-2016-toan-bo-120-cum-thi-da-co-diem.html",
-    examples: ["17006021", "Nguyễn Thị Hoa"],
+    examples: ["TKG002747", "Nguyễn Bửu Lộc"],
     presets: PRESETS_2016,
   },
   2017: {