From 08a408ca96693a146161e6fbb7c3fbaa6e855758 Mon Sep 17 00:00:00 2001 From: tiennm99 Date: Thu, 13 Aug 2026 22:54:50 +0700 Subject: [PATCH] docs(plans): drop the completed migration plans MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both plans are done and merged, and their contents now describe a tree that no longer exists — go-parser/, four datasets, npm entry points, a Rust crate. Git history keeps them. The parity evidence stays: docs/data-pipeline.md cites parser-parity-result.md for the recovered foreign-language scores, so that report and the two JSON snapshots behind it remain, now marked as an archived record. --- ...e-01-standard-schema-and-unified-parser.md | 170 ----------- .../phase-02-repo-restructure.md | 149 ---------- ...3-unified-frontend-and-dataset-registry.md | 249 ---------------- .../phase-04-build-and-deploy-pipeline.md | 210 ------------- .../phase-05-parity-verification-and-docs.md | 128 -------- .../plan.md | 184 ------------ .../phase-01-scaffold-and-biff-reader-gate.md | 279 ------------------ .../phase-02-schema-and-config.md | 195 ------------ .../phase-03-transform-core.md | 160 ---------- .../phase-04-reader-writer-and-cli.md | 210 ------------- .../phase-05-2016-format-detection.md | 146 --------- .../phase-06-differential-parity-gate.md | 220 -------------- .../phase-07-ci-docs-and-script-port.md | 269 ----------------- plans/260813-1502-go-parser-migration/plan.md | 193 ------------ ...go-parser-side-by-side-migration-report.md | 79 ----- ...1502-rust-to-go-parser-migration-report.md | 70 ----- plans/reports/parser-parity-result.md | 8 +- 17 files changed, 7 insertions(+), 2912 deletions(-) delete mode 100644 plans/260813-0956-unify-frontend-standard-schema/phase-01-standard-schema-and-unified-parser.md delete mode 100644 plans/260813-0956-unify-frontend-standard-schema/phase-02-repo-restructure.md delete mode 100644 plans/260813-0956-unify-frontend-standard-schema/phase-03-unified-frontend-and-dataset-registry.md delete mode 100644 plans/260813-0956-unify-frontend-standard-schema/phase-04-build-and-deploy-pipeline.md delete mode 100644 plans/260813-0956-unify-frontend-standard-schema/phase-05-parity-verification-and-docs.md delete mode 100644 plans/260813-0956-unify-frontend-standard-schema/plan.md delete mode 100644 plans/260813-1502-go-parser-migration/phase-01-scaffold-and-biff-reader-gate.md delete mode 100644 plans/260813-1502-go-parser-migration/phase-02-schema-and-config.md delete mode 100644 plans/260813-1502-go-parser-migration/phase-03-transform-core.md delete mode 100644 plans/260813-1502-go-parser-migration/phase-04-reader-writer-and-cli.md delete mode 100644 plans/260813-1502-go-parser-migration/phase-05-2016-format-detection.md delete mode 100644 plans/260813-1502-go-parser-migration/phase-06-differential-parity-gate.md delete mode 100644 plans/260813-1502-go-parser-migration/phase-07-ci-docs-and-script-port.md delete mode 100644 plans/260813-1502-go-parser-migration/plan.md delete mode 100644 plans/reports/from-brainstorm-to-plan-260813-1502-go-parser-side-by-side-migration-report.md delete mode 100644 plans/reports/from-scout-to-brainstorm-260813-1502-rust-to-go-parser-migration-report.md diff --git a/plans/260813-0956-unify-frontend-standard-schema/phase-01-standard-schema-and-unified-parser.md b/plans/260813-0956-unify-frontend-standard-schema/phase-01-standard-schema-and-unified-parser.md deleted file mode 100644 index 761f9f5..0000000 --- a/plans/260813-0956-unify-frontend-standard-schema/phase-01-standard-schema-and-unified-parser.md +++ /dev/null @@ -1,170 +0,0 @@ ---- -phase: 1 -title: "Standard schema and unified parser" -status: completed -priority: P1 -dependencies: [] -effort: "" ---- - -# Phase 1: Standard schema and unified parser - -## Overview - -Move the SQL schema out of the four TOML configs into one Rust module, widen it -to the 22-column union, and prove the four rebuilt databases still match their -current contents. Done in place under the existing `2016/`/`2017/` layout so -parity is measurable before anything moves. - -## Context - -`2016/tools/xlsxread` and `2017/tools/xlsxread` are the same crate. Verified by -`diff -rq`: `cli.rs`, `error.rs`, `reader.rs` are byte-identical; `audit.rs`, -`config.rs`, `lib.rs`, `main.rs`, `transform.rs`, `writer.rs` differ only where -2016 adds a branch; `format_detect_2016.rs` (549 lines) exists only in 2016. -The 2016 crate is therefore the base to keep — it already builds 2017 datasets -(its `configs/` holds 2017 fixtures used by `tests/golden.rs`). - -Current per-dataset schema divergence: - -| Column group | 2016 | 2017 / old / old2 | -|---|---|---| -| `ten_cum_thi`, `gioi_tinh` | present | absent | -| `khtn`, `khxh`, `gdcd`, `tieng_nga` | absent | present | -| `tieng_duc`, `tieng_nhat` | present | absent | - -## Requirements - -**Functional** -- One DDL, one INSERT, one ordered subject list, one regex map — used by all four datasets -- Configs keep only genuinely per-dataset settings -- Rebuilt DBs preserve every value currently produced - -**Non-functional** -- No measurable build-time regression (currently a few minutes per dataset) -- DB size growth from the added NULL columns stays under ~2% - -## Architecture - -**New `parser/src/schema.rs`** (written into `2016/tools/xlsxread/src/` this -phase; relocated in Phase 2) exports: - -- `pub const DDL: &str` — the 22-column CREATE TABLE + 3 indexes -- `pub const INSERT_SQL: &str` — 22 positional placeholders -- `pub const IDENTITY_FIELDS: &[&str]` — 6 identity columns in INSERT order -- `pub const SCORE_FIELDS: &[&str]` — 16 subject columns in INSERT order -- `pub fn score_patterns() -> Vec<(&'static str, &'static str)>` — the union of - all 16 subject regexes (2016's 12 ∪ 2017's 14; both sets share 10) - -**`writer.rs` collapses.** `SCORE_FIELDS_2016`, `SCORE_FIELDS_2017`, the -`SCORE_FIELDS` alias, and `insert_row_2016` all disappear. A single `insert_row` -binds identity fields then subject fields, always 22 params. - -**`config.rs` shrinks.** `[schema]` and `[insert]` sections are removed from -`DatasetConfig`; `[scores]` becomes optional and unused (delete the field once -no config sets it). `columns` stays `Option` — the 2016 detection path -does not use it. - -**`main.rs`** keeps the `run_build_2016` / `run_build_standard` split; both now -call the same `insert_row` with `schema::SCORE_FIELDS`. - -### Union regex risk — RESOLVED, and it found a real bug - -Applying all 16 patterns to every dataset means a source cell containing a -subject the old config never looked for now populates a column it previously -could not. - -**Outcome: the gate fired, and the matches were real data, not false positives.** - -The pre-refactor configs listed 12 subject regexes (2016) and 14 (2017). Neither -list was complete — Vietnamese candidates could sit German, Japanese and Russian -in *both* exam years. Every affected student previously ended up with **no** -foreign-language score at all. Unifying to 16 patterns recovered 1,691 scores: - -| Dataset | Recovered | -|---|---| -| 2016 | 182 × `tieng_nga` | -| 2017 | 93 × `tieng_duc`, 512 × `tieng_nhat` | -| 2017-old | 85 × `tieng_duc`, 484 × `tieng_nhat` | -| 2017-old2 | 22 × `tieng_duc`, 313 × `tieng_nhat` | - -Verified genuine, not spurious: -- Across all four datasets every student holds **zero or exactly one** foreign - language — never two. So the new columns duplicate nothing. -- Each affected student had *all* language columns NULL beforehand - (e.g. SBD `01003198`: all NULL → `tieng_duc = 8`). -- Counts track dataset size consistently across the three 2017 generations - (93/85/22 German, 512/484/313 Japanese). -- Score ranges and distributions are ordinary exam values (0–10). - -Row counts, all 18 pre-existing per-column non-NULL counts, and the -deterministic student samples were **identical** — nothing was lost. - -Accepted as a data-quality fix. The earlier guidance to add a per-config subject -allowlist assumed spurious matches and does **not** apply; suppressing these -would knowingly discard real scores. The exact counts are now encoded as -`APPROVED_RECOVERY` in `verify-parity.js`, so any *other* newly-populated column -— or any drift in these numbers — still fails the gate. - -## Related Code Files - -- Create: `2016/tools/xlsxread/src/schema.rs` -- Modify: `2016/tools/xlsxread/src/writer.rs` (drop dual insert paths + field lists) -- Modify: `2016/tools/xlsxread/src/config.rs` (drop `[schema]`/`[insert]`/`[scores]`) -- Modify: `2016/tools/xlsxread/src/main.rs` (single insert call site per path) -- Modify: `2016/tools/xlsxread/src/lib.rs` (declare `schema` module) -- Modify: `2016/tools/xlsxread/src/transform.rs` (source patterns from `schema`) -- Modify: `2016/tools/xlsxread/tests/golden.rs` (assert against canonical schema) -- Create: `2016/tools/xlsxread/configs/thptqg2017-data-old2.toml` (copy from 2017 crate) -- Modify: all 4 configs — strip DDL/INSERT/scores down to parse rules -- Delete (Phase 2): `2017/tools/` - -## Implementation Steps - -1. **Capture the baseline first.** Build all four DBs with the *current* code and - record, per dataset: `SELECT COUNT(*) FROM student`, and for each column - `SUM(col IS NOT NULL)`. Store as `plans/reports/parser-parity-baseline.json`. - Nothing else in this plan is verifiable without this artifact. -2. Copy `thptqg2017-data-old2.toml` into the 2016 crate's `configs/`. -3. Write `schema.rs` with DDL, INSERT, field orders, and the union regex map. -4. Rewrite `writer.rs` to a single `insert_row`; delete `insert_row_2016` and - the three score-field constants. -5. Strip `[schema]`, `[insert]`, `[scores]` from all four configs; update - `config.rs` structs to match. Each config should end at ~15–20 lines. -6. Update `main.rs` and `transform.rs` call sites. -7. Update `tests/golden.rs` to assert the canonical column set. -8. `cargo test` — golden tests must pass. -9. Rebuild all four DBs; regenerate the same stats and diff against the baseline. - -## Tests / Validation - -- `cargo test` — 63 tests (55 unit + 8 golden) -- `cargo clippy --all-targets -- -D warnings` -- Row count per dataset equals baseline exactly -- Per-column non-NULL count equals baseline for every previously-existing column -- Newly-added columns read 0 non-NULL, except the approved recoveries above -- DB file size within 2% of baseline - -**Note on the clippy gate:** it was already red before this phase — measured at -**8 errors on the branch base**. This phase's changes introduced none. The -pre-existing lints were cleared in a separate commit so the gate is genuinely -green from here on. - -## Success Criteria - -- [x] `schema.rs` is the only place DDL/INSERT/subject-order/regexes are written -- [x] All 4 configs under 35 lines, containing no SQL -- [x] `writer.rs` has exactly one insert function -- [x] `cargo test` (63 passing) and `cargo clippy --all-targets -D warnings` green -- [x] All 4 DBs match baseline row counts and per-column non-NULL counts -- [x] Baseline JSON committed under `plans/reports/` -- [x] Config parsing rejects leftover SQL sections (`deny_unknown_fields`) - -## Risk Assessment - -| Risk | Mitigation | -|---|---| -| Union regexes populate unexpected columns | Baseline diff catches it; fall back to per-config subject allowlist | -| INSERT param order drifts from DDL order | Single `IDENTITY_FIELDS`/`SCORE_FIELDS` source drives both; golden test asserts round-trip | -| Rebuilding 419 MB of source is slow | Run once per verification pass, not per edit; iterate against golden fixtures | -| Baseline skipped under time pressure | Phase 1 is unverifiable without it — treat step 1 as blocking | diff --git a/plans/260813-0956-unify-frontend-standard-schema/phase-02-repo-restructure.md b/plans/260813-0956-unify-frontend-standard-schema/phase-02-repo-restructure.md deleted file mode 100644 index 1665226..0000000 --- a/plans/260813-0956-unify-frontend-standard-schema/phase-02-repo-restructure.md +++ /dev/null @@ -1,149 +0,0 @@ ---- -phase: 2 -title: "Repo restructure" -status: completed -priority: P1 -dependencies: [1] -effort: "" ---- - -# Phase 2: Repo restructure - -## Overview - -Move files into the target layout and migrate pnpm → npm. Relocation plus -package-manager swap — no application logic changes. Kept as its own commit so -the 419 MB data move is trivially revertable and reviewable separately from -behavior changes. - -## Requirements - -- All moves via `git mv` so blobs are reused and history follows -- Working tree builds after the move (paths updated, nothing dangling) -- Repository size does not grow -- npm is the only package manager; `package-lock.json` committed, no pnpm files remain - -## Architecture - -Git stores blobs content-addressed, so renaming 299 tracked data files -(~419 MB working-tree) adds no new objects. The cost is local I/O and one large -tree rewrite, not repository growth. - -Data directory sizes being moved: `2016/data` 62 MB, `2017/data` 286 MB, -`2017/data-old` 39 MB, `2017/data-old2` 32 MB. - -### pnpm → npm - -A single package with 3 runtime and 9 dev dependencies, no workspace. pnpm's -advantages (strict resolution, shared store) buy nothing at this size, and two -pieces of scaffolding disappear with it: - -- `pnpm-workspace.yaml` exists **only** to whitelist `better-sqlite3`'s native - build (`allowBuilds`). npm runs postinstall by default, so the file has no npm - equivalent — it is deleted, not translated. Verified: this is the file's - entire content in both projects. -- The `pnpm/action-setup@v4` CI step and `cache: 'pnpm'` both drop out. - -Lockfiles cannot be converted; `package-lock.json` is generated fresh from -`package.json`. Migration direction is safe: pnpm's strict `node_modules` layout -forbids phantom dependencies, so anything that resolved under pnpm also resolves -under npm's flat tree. The reverse would not hold. - -Latent issue surfaced while checking: `2017/scripts/diff-datasets.js` imports -`better-sqlite3`, which is **not declared in any `package.json`** — that script -cannot run today without an ad-hoc install. Do not add the dependency; Phase 5 -uses the built-in `node:sqlite` instead (verified working on Node 24, no flag). -Either port `diff-datasets.js` to `node:sqlite` in this phase or leave it broken -exactly as it is today and note it — do not silently half-fix it. - -## Related Code Files - -**Moves** - -| From | To | -|---|---| -| `2016/tools/xlsxread/` | `parser/` | -| `2016/tools/xlsxread/configs/thptqg2016-data.toml` | `parser/configs/2016.toml` | -| `2017/tools/xlsxread/configs/thptqg2017-data.toml` | `parser/configs/2017.toml` | -| `2017/tools/xlsxread/configs/thptqg2017-data-old.toml` | `parser/configs/2017-old.toml` | -| `2017/tools/xlsxread/configs/thptqg2017-data-old2.toml` | `parser/configs/2017-old2.toml` | -| `2017/scripts/` | `parser/scripts/` | -| `2016/data/` | `data/2016/` | -| `2017/data/` | `data/2017/` | -| `2017/data-old/` | `data/2017-old/` | -| `2017/data-old2/` | `data/2017-old2/` | -| `2017/src/` | `src/` | -| `2017/index.html` | `index.html` (overwrites the old static landing page) | -| `2017/package.json`, `eslint.config.js`, `vite.config.js` | repo root | -| `2016/docs/*`, `2017/docs/*` | `docs/` | -| `2017/LICENSE` | `LICENSE` | - -The old root `index.html` (the static hub) is **not moved** — it becomes -`src/components/hub.jsx` in Phase 3. Read it before deleting; its four links, -candidate counts, and Vietnamese copy are the source material for that -component. - -**Deletes** -- `2016/` and `2017/` directories entirely (after moves) -- `2016/src/` — superseded by `src/` (its unique columns and SQL presets are ported in Phase 3) -- `2016/tools/xlsxread/configs/thptqg2017-*.toml` — test fixtures, superseded by real configs -- `2016/pnpm-lock.yaml`, `2017/pnpm-lock.yaml`, both `pnpm-workspace.yaml` -- Duplicate `2016/package.json`, `2016/eslint.config.js`, `2016/vite.config.js`, `2016/LICENSE` -- Root `index.html` (static hub) — only after its content is captured for `hub.jsx` - -**Merges** -- `.gitignore` — one root file. 2016's is the verbose GitHub Node template, - 2017's is terse and accurate. Take 2017's as the base, add `parser/target/`, - `.build/`, and `dist/`. -- `package.json` — root file keeps 2017's dependency set (identical to 2016's). - Drop the `"packageManager": "pnpm@11.1.1"` field. - -## Implementation Steps - -1. Branch: `refactor/unify-frontend-and-schema`. -2. Copy the old root `index.html` content somewhere durable for Phase 3 - (`plans/reports/` or the phase-03 file itself) — it is the hub's source copy. -3. `git mv 2017/src src`, then the 2017 root config files, then - `git mv 2017/index.html index.html`. -4. `git mv 2016/tools/xlsxread parser`, then rename the four configs. -5. `git mv 2017/scripts parser/scripts`. -6. Move the four data directories. -7. Merge docs into `docs/`, prefixing 2016-specific filenames where they collide - (`deployment-guide.md` and `system-architecture.md` exist in both — read both - before merging; they describe different pipelines). -8. Delete the emptied `2016/` and `2017/` trees plus both `pnpm-workspace.yaml` - and both `pnpm-lock.yaml`. -9. Write the merged root `.gitignore`. -10. Drop `"packageManager"` from `package.json`; run `npm install` to generate - `package-lock.json`; commit the lockfile. -11. Fix paths inside moved files: `parser/Cargo.toml` package name/paths, - config `--input`/`--output` defaults, `parser/scripts/*.js` relative paths. -12. Replace `pnpm` with `npm run` in every `package.json` script body. -13. `cargo test` from `parser/` — must still be green. - -## Tests / Validation - -- `cargo test --manifest-path parser/Cargo.toml` -- `npm ci && npm run lint` at root — `npm ci` proves the lockfile is in sync -- `git status` shows renames (R), not delete+add pairs -- No file outside `plans/` still references `2016/tools`, `2017/tools`, - `2017/src`, `2017/data`, or `pnpm` — grep to confirm - -## Success Criteria - -- [x] Target layout matches `plan.md` exactly -- [x] `2016/` and `2017/` no longer exist -- [x] Old hub page content captured before deletion -- [x] Git reports renames, repository size unchanged -- [x] `package-lock.json` committed; `npm ci` succeeds; zero pnpm files remain -- [x] `cargo test` green from the new location - -## Risk Assessment - -| Risk | Mitigation | -|---|---| -| Old hub page deleted before its copy is captured | Step 2 runs before any move; content also recoverable from git history | -| Docs collide silently on merge | Read both copies before merging; two same-named files describe different pipelines | -| Git records delete+add instead of rename | Use `git mv`; verify with `git status` before commit | -| Stale path references in scripts/CI | Grep sweep in validation; Phase 4 rewrites the workflow | -| Fresh npm lockfile resolves different transitive versions than pnpm did | All deps are caret-ranged and already floating; `npm run lint` + the Phase 4 local build run are the check | diff --git a/plans/260813-0956-unify-frontend-standard-schema/phase-03-unified-frontend-and-dataset-registry.md b/plans/260813-0956-unify-frontend-standard-schema/phase-03-unified-frontend-and-dataset-registry.md deleted file mode 100644 index e36e839..0000000 --- a/plans/260813-0956-unify-frontend-standard-schema/phase-03-unified-frontend-and-dataset-registry.md +++ /dev/null @@ -1,249 +0,0 @@ ---- -phase: 3 -title: "Unified frontend and dataset registry" -status: completed -priority: P1 -dependencies: [2] -effort: "" ---- - -# Phase 3: Unified frontend and dataset registry - -## Overview - -Make the single 2017-derived frontend serve every page on the site: the four -dataset views plus the `/thptqg/` hub. Three kinds of work — port 2016's only -unique feature (cluster + gender columns) into the shared components, extract -everything genuinely per-dataset into a runtime registry, and add a small -pathname router with a hub route. - -## Context — what actually differs - -The 2017 frontend is a superset in every respect except two columns. Verified by -reading both trees: - -**2017 has, 2016 lacks:** debounced live search with input-mode hints -(`search-form.jsx`, 127 vs 46 lines), `?q=` URL deep links, `student-detail.jsx` -(183 lines, single-result view), `lib/admission-blocks.js` (49 blocks + -`scoreTier` ladder), all-NULL column hiding, `/` and Ctrl+Enter shortcuts, -footer total count, load progress bar, richer SQL presets (335 vs 219 lines). - -**2016 has, 2017 lacks:** `ten_cum_thi` and `gioi_tinh` table columns with a -`cumthi-cell` title tooltip, and a simpler 3-tier `scoreClass` (superseded by -2017's 6-tier `scoreTier`). - -## Requirements - -**Functional** -- 2016 site keeps cluster + gender columns; 2017 sites must not show them -- 2016 site gains every 2017 feature listed above -- Per-dataset chrome (title, subtitle, source, DB size, search examples, SQL presets) is data, not code -- Admission-block computation works for both exam years without branching -- `/thptqg/` renders a hub listing the four datasets; no DB is fetched there -- Deep links (`?q=`) keep working on every dataset route - -**Non-functional** -- Dataset identity resolves from `location.pathname` — no fetch, no env plumbing -- No routing library; five static routes do not justify a dependency -- Hub route must not pull the 47 MB DB into its critical path - -## Architecture - -### `src/datasets.js` — the registry - -```js -export const DATASETS = [ - { - id: "2016", // === URL segment === data dir === config === db file - dbSizeMb: 48, - title: "Tra cứu điểm thi THPT Quốc gia 2016", - subtitle: "Dữ liệu thí sinh toàn quốc · Hỗ trợ truy vấn SQL tùy chỉnh", - source: "Bộ GD&ĐT", - examples: ["", "Nguyễn Minh Tiến"], - presets: PRESETS_2016, // from 2016/src/components/custom-query.jsx - blocks: BLOCKS_2016, // see admission blocks below - }, - { id: "2017", /* presets: PRESETS_2017 */ }, - { id: "2017-old", /* same presets as 2017, own label */ }, - { id: "2017-old2", /* same presets as 2017, own label */ }, -]; - -export const pathOf = (d) => `${import.meta.env.BASE_URL}${d.id}/`; -export const dbOf = (d) => `${import.meta.env.BASE_URL}db/${d.id}.db.gz`; -``` - -Plain runtime data — no `import.meta.env.VITE_DATASET` inlining, no build -variants. All four entries ship in the one bundle; the preset SQL totals a few -KB gzipped. Because URLs are flat, path and DB URL are *derived* from `id` -rather than stored, so a dataset cannot be misconfigured into pointing at the -wrong database. - -### `src/router.js` - -Flat URLs make this an exact match on a single segment — the segment **is** the -dataset ID: - -```js -export function resolveRoute(pathname = location.pathname) { - const seg = pathname - .slice(import.meta.env.BASE_URL.length) // strip "/thptqg/" - .replace(/\/$/, ""); - return DATASETS.find((d) => d.id === seg) ?? null; // null → hub -} -``` - -No prefix sorting, no ambiguity between `2017` and `2017-old` — that entire -class of bug is designed out by the flat scheme. No `react-router`. - -### Legacy path redirects - -The two old nested URLs are kept alive by a small map, so existing links and -bookmarks resolve instead of 404ing: - -```js -const LEGACY = { "2017/old": "2017-old", "2017/old2": "2017-old2" }; -``` - -On a legacy match, `history.replaceState` to the canonical flat URL **preserving -`location.search`** (the `?q=` deep link must survive), then resolve normally. -Phase 4 emits `index.html` at both legacy paths so Pages serves them at all. - -Droppable if you'd rather let the old URLs 404 — it costs ~5 lines and two file -copies, and nothing else in the plan depends on it. - -### `src/components/hub.jsx` - -Renders the four dataset links from `DATASETS` via `pathOf()`. Content ported -from the old static root `index.html` (captured in Phase 2 step 2): heading, the -sql.js one-liner explanation, candidate counts, and the "phiên bản cũ" grouping -of the two 2017 variants. Styled with the app's existing CSS instead of the old -inline `