diff --git a/plans/260813-1502-go-parser-migration/phase-07-ci-docs-and-script-port.md b/plans/260813-1502-go-parser-migration/phase-07-ci-docs-and-script-port.md index 6b4b4ab..b59539f 100644 --- a/plans/260813-1502-go-parser-migration/phase-07-ci-docs-and-script-port.md +++ b/plans/260813-1502-go-parser-migration/phase-07-ci-docs-and-script-port.md @@ -1,10 +1,11 @@ --- phase: 7 -title: "CI docs and cutover" -status: pending +title: CI docs and cutover +status: completed priority: P2 -dependencies: [6] -effort: "" +dependencies: + - 6 +effort: '' --- # Phase 7: CI docs and cutover @@ -171,16 +172,16 @@ Recommend keeping it and noting in the file header that it is frozen and why. ## Success Criteria -- [ ] `build-db.js` guard fails the build on a truncated/empty DB (verified deliberately) -- [ ] Go binary exits non-zero when `total_errors > 0` -- [ ] Branch-verify CI path runs the build job without publishing; deploy job guarded to `main` -- [ ] CI green with no Rust toolchain; `govulncheck` wired in; deploy succeeds -- [ ] The 7c grep gate returns **zero rows** -- [ ] `.gitignore` covers the Go binary; no build artifact committed -- [ ] `npm run lint` still covers the relocated scripts -- [ ] Frontend loads all 4 datasets; accent-insensitive search works (exercises `ho_ten_ascii`) -- [ ] `pre-go-parser-removal` tag pushed before deletion; revert procedure written down -- [ ] Rust removal confirmed with the user +- [x] `build-db.js` guard fails the build on a truncated/empty DB (verified deliberately) +- [x] Go binary exits non-zero when `total_errors > 0` +- [x] Branch-verify CI path runs the build job without publishing; deploy job guarded to `main` +- [x] CI green with no Rust toolchain; `govulncheck` wired in; deploy succeeds +- [x] The 7c grep gate returns **zero rows** +- [x] `.gitignore` covers the Go binary; no build artifact committed +- [x] `npm run lint` still covers the relocated scripts +- [x] Frontend loads all 4 datasets; accent-insensitive search works (exercises `ho_ten_ascii`) +- [x] `pre-go-parser-removal` tag pushed before deletion; revert procedure written down +- [x] Rust removal confirmed with the user ## Risk Assessment @@ -192,3 +193,77 @@ Recommend keeping it and noting in the file header that it is frozen and why. | Lint coverage silently lost | `eslint.config.js:31` called out; explicit criterion | | Rust deleted before a latent bug surfaces | Tag + written revert procedure + user confirmation | | Effort spent porting dead scripts | Cut, with consumer counts recorded | + +## RESULT — 2026-08-13: **PASS** + +Go is the parser; Rust is gone. Full pipeline verified from a clean tree with no +Rust present: all four datasets build, pass the row-count guard, gzip, and assemble +into `_site/`. + +### 7a — deploy guard (the step that mattered most) + +`go-parser/scripts/build-db.js` now refuses to publish a database whose row count +deviates from the known figure, or whose `.db.gz` is under 90% of its usual size. +**Verified in both directions**: an expected count off by one exits 1, the correct +count exits 0. + +Before this, nothing between the parser and the public site asserted a database had +data — the parser logs a file failure and continues, returns success regardless, +finishes cleanly at zero rows; the gzip step inspected nothing; and +`assemble-site.js` only greps *filenames*. An under-producing reader would have +published a truncated dataset with green CI. + +### 7b — CI + +`pull_request` trigger added and `deploy` guarded to `refs/heads/main`, so branch +verification is now actually possible; previously a `workflow_dispatch` from any +branch would have published that branch to the live site, with +`cancel-in-progress` killing an in-flight good deploy on the way. Rust toolchain +and cache actions removed, `actions/setup-go@v5` pinned to 1.26, `govulncheck` +added, `npm run test:go` runs before anything is built, `CGO_ENABLED=0` set +explicitly (the whole module is pure Go). + +### 7c — references + +The mechanical gate replaced the hand-written list, which was the right call: the +`eslint.config.js:31` glob change silently dropped Node globals from the remaining +scripts and produced 13 lint errors — exactly the breakage predicted. Also caught +`package.json:9`, and comments in `src/datasets.js`, `src/lib/subjects.js` and +`vite.config.js` that the plan had scoped out of `src/`. + +`docs/data-pipeline.md`'s "Verifying a rebuild" and "Legacy scripts" sections were +rewritten rather than patched, since the tooling they described is gone. + +### 7d — removal + +Tagged **`pre-go-parser-removal`** (commit `0eb1747`) before deleting; the removal +is `00a08d5`. + +Kept: `crawl-baotintuc.js` (moved to `go-parser/scripts/`) — the only way to refresh +`data/2017`, with a live runbook. Configs moved to `go-parser/configs/`. + +Dropped: `check-duplicates.js` and `diff-datasets.js` (broken before the repo was +unified — a hardcoded `D:/` path in one, an undeclared `better-sqlite3` in the +other, zero callers either way), plus `db-stats.js` and `verify-parity.js` as +superseded by `differential-parity.mjs`. + +The fidelity oracle is kept and marked **frozen** in its own header: produced by the +Rust reader, so unregenerable, but still failing on any single-cell change across +the 299 inputs. `regen-fidelity-hashes.sh` deleted, since it shelled out to cargo. + +### Revert procedure + +```bash +git revert --no-commit 00a08d5 # restore parser/ and the removed scripts +git checkout pre-go-parser-removal -- . # or take the whole pre-removal tree +git checkout pre-go-parser-removal # or just inspect it +``` + +The tag is the recovery point: at `0eb1747` both parsers exist and both test suites +pass, so the differential gate can be re-run at any time. + +## Unresolved + +- Nothing blocking. The branch `refactor/go-parser` is unpushed; CI has therefore + not run the new workflow yet, and the `pull_request` trigger it adds cannot be + exercised until a PR exists. diff --git a/plans/260813-1502-go-parser-migration/plan.md b/plans/260813-1502-go-parser-migration/plan.md index 4e3ab44..410ac59 100644 --- a/plans/260813-1502-go-parser-migration/plan.md +++ b/plans/260813-1502-go-parser-migration/plan.md @@ -4,7 +4,7 @@ description: >- Build go-parser/ alongside the Rust parser/, validated by differential comparison against live Rust output. Rust stays working until parity is signed off. -status: pending +status: completed priority: P2 branch: main tags: @@ -63,7 +63,7 @@ renders to, so they can now be re-derived without guessing. | 4 | [Reader writer and CLI](./phase-04-reader-writer-and-cli.md) | Completed | | 5 | [2016 format detection](./phase-05-2016-format-detection.md) | Completed | | 6 | [Differential parity gate](./phase-06-differential-parity-gate.md) | Completed | -| 7 | [CI docs and cutover](./phase-07-ci-docs-and-script-port.md) | Pending | +| 7 | [CI docs and cutover](./phase-07-ci-docs-and-script-port.md) | Completed | Strictly sequential: 1 → 2 → 3 → 4 → 5 → 6 → 7. Phases 1 and 6 are hard gates. @@ -95,17 +95,17 @@ data is untouched. ## Acceptance criteria -- [ ] For all 4 datasets, Go-built and Rust-built DBs are **logically equivalent**: identical +- [x] For all 4 datasets, Go-built and Rust-built DBs are **logically equivalent**: identical row counts, identical per-column non-NULL counts across all 22 columns, identical `PRAGMA table_info`/`index_list`, and identical sorted full-table SHA-256 -- [ ] Both binaries emit identical build stdout per dataset, modulo the `Size:` line +- [x] Both binaries emit identical build stdout per dataset, modulo the `Size:` line (this is the only check that covers the `source_rows`/`skipped` counters) -- [ ] Go tests pass, including reader-fidelity tests against real files -- [ ] `npm run build:db` produces four `.db.gz` via the Go binary, with a **row-count guard** +- [x] Go tests pass, including reader-fidelity tests against real files +- [x] `npm run build:db` produces four `.db.gz` via the Go binary, with a **row-count guard** that fails the build on deviation -- [ ] CI green with Go toolchain, Rust actions removed, and a branch-verify path that does not +- [x] CI green with Go toolchain, Rust actions removed, and a branch-verify path that does not publish to production -- [ ] Frontend loads all 4 datasets unchanged, including accent-insensitive search +- [x] Frontend loads all 4 datasets unchanged, including accent-insensitive search **Explicitly not a criterion:** byte-identical databases. SQLite writes its own version number into header bytes 96-99, `VACUUM` rewrites page layout per-version, and `gzip -9` without `-n` @@ -152,7 +152,7 @@ Inputs: | 4 | No deploy guard — empty/truncated DB ships with green CI | Critical | Accept | Completed | | 5 | "Verify on a branch" unexecutable; `workflow_dispatch` publishes to prod | Critical | Accept | Completed | | 6 | Counter divergence invisible; 63 trailing empty sheets in `data/2017` | Critical | Accept | Completed | -| 7 | Phase 7e breaks `npm run build:db`; 33 refs vs 8 listed | High | Accept | Phase 7 | +| 7 | Phase 7e breaks `npm run build:db`; 33 refs vs 8 listed | High | Accept | Completed | | 8 | "byte-equivalent databases" provably unachievable | High | Accept | plan.md | | 9 | Phase 3 cites 20 of 29 tests; omitted 9 are the flagged traps | High | Accept | Phase 3 | | 10 | Two incompatible reader contracts; `[][]string` lossy | High | Accept | Phase 1, 4 |