mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-10-15 22:15:13 +00:00
docs(plans): record the Go parser migration outcome
This commit is contained in:
1 parent
00a08d5fab
commit
8530df0154
2 files changed
+98
-23
No files matched your search
@@ -1,10 +1,11 @@
|
||||
---
|
||||
phase: 7
|
||||
title: "CI docs and cutover"
|
||||
status: pending
|
||||
title: CI docs and cutover
|
||||
status: completed
|
||||
priority: P2
|
||||
dependencies: [6]
|
||||
effort: ""
|
||||
dependencies:
|
||||
- 6
|
||||
effort: ''
|
||||
---
|
||||
|
||||
# Phase 7: CI docs and cutover
|
||||
@@ -171,16 +172,16 @@ Recommend keeping it and noting in the file header that it is frozen and why.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `build-db.js` guard fails the build on a truncated/empty DB (verified deliberately)
|
||||
- [ ] Go binary exits non-zero when `total_errors > 0`
|
||||
- [ ] Branch-verify CI path runs the build job without publishing; deploy job guarded to `main`
|
||||
- [ ] CI green with no Rust toolchain; `govulncheck` wired in; deploy succeeds
|
||||
- [ ] The 7c grep gate returns **zero rows**
|
||||
- [ ] `.gitignore` covers the Go binary; no build artifact committed
|
||||
- [ ] `npm run lint` still covers the relocated scripts
|
||||
- [ ] Frontend loads all 4 datasets; accent-insensitive search works (exercises `ho_ten_ascii`)
|
||||
- [ ] `pre-go-parser-removal` tag pushed before deletion; revert procedure written down
|
||||
- [ ] Rust removal confirmed with the user
|
||||
- [x] `build-db.js` guard fails the build on a truncated/empty DB (verified deliberately)
|
||||
- [x] Go binary exits non-zero when `total_errors > 0`
|
||||
- [x] Branch-verify CI path runs the build job without publishing; deploy job guarded to `main`
|
||||
- [x] CI green with no Rust toolchain; `govulncheck` wired in; deploy succeeds
|
||||
- [x] The 7c grep gate returns **zero rows**
|
||||
- [x] `.gitignore` covers the Go binary; no build artifact committed
|
||||
- [x] `npm run lint` still covers the relocated scripts
|
||||
- [x] Frontend loads all 4 datasets; accent-insensitive search works (exercises `ho_ten_ascii`)
|
||||
- [x] `pre-go-parser-removal` tag pushed before deletion; revert procedure written down
|
||||
- [x] Rust removal confirmed with the user
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
@@ -192,3 +193,77 @@ Recommend keeping it and noting in the file header that it is frozen and why.
|
||||
| Lint coverage silently lost | `eslint.config.js:31` called out; explicit criterion |
|
||||
| Rust deleted before a latent bug surfaces | Tag + written revert procedure + user confirmation |
|
||||
| Effort spent porting dead scripts | Cut, with consumer counts recorded |
|
||||
|
||||
## RESULT — 2026-08-13: **PASS**
|
||||
|
||||
Go is the parser; Rust is gone. Full pipeline verified from a clean tree with no
|
||||
Rust present: all four datasets build, pass the row-count guard, gzip, and assemble
|
||||
into `_site/`.
|
||||
|
||||
### 7a — deploy guard (the step that mattered most)
|
||||
|
||||
`go-parser/scripts/build-db.js` now refuses to publish a database whose row count
|
||||
deviates from the known figure, or whose `.db.gz` is under 90% of its usual size.
|
||||
**Verified in both directions**: an expected count off by one exits 1, the correct
|
||||
count exits 0.
|
||||
|
||||
Before this, nothing between the parser and the public site asserted a database had
|
||||
data — the parser logs a file failure and continues, returns success regardless,
|
||||
finishes cleanly at zero rows; the gzip step inspected nothing; and
|
||||
`assemble-site.js` only greps *filenames*. An under-producing reader would have
|
||||
published a truncated dataset with green CI.
|
||||
|
||||
### 7b — CI
|
||||
|
||||
`pull_request` trigger added and `deploy` guarded to `refs/heads/main`, so branch
|
||||
verification is now actually possible; previously a `workflow_dispatch` from any
|
||||
branch would have published that branch to the live site, with
|
||||
`cancel-in-progress` killing an in-flight good deploy on the way. Rust toolchain
|
||||
and cache actions removed, `actions/setup-go@v5` pinned to 1.26, `govulncheck`
|
||||
added, `npm run test:go` runs before anything is built, `CGO_ENABLED=0` set
|
||||
explicitly (the whole module is pure Go).
|
||||
|
||||
### 7c — references
|
||||
|
||||
The mechanical gate replaced the hand-written list, which was the right call: the
|
||||
`eslint.config.js:31` glob change silently dropped Node globals from the remaining
|
||||
scripts and produced 13 lint errors — exactly the breakage predicted. Also caught
|
||||
`package.json:9`, and comments in `src/datasets.js`, `src/lib/subjects.js` and
|
||||
`vite.config.js` that the plan had scoped out of `src/`.
|
||||
|
||||
`docs/data-pipeline.md`'s "Verifying a rebuild" and "Legacy scripts" sections were
|
||||
rewritten rather than patched, since the tooling they described is gone.
|
||||
|
||||
### 7d — removal
|
||||
|
||||
Tagged **`pre-go-parser-removal`** (commit `0eb1747`) before deleting; the removal
|
||||
is `00a08d5`.
|
||||
|
||||
Kept: `crawl-baotintuc.js` (moved to `go-parser/scripts/`) — the only way to refresh
|
||||
`data/2017`, with a live runbook. Configs moved to `go-parser/configs/`.
|
||||
|
||||
Dropped: `check-duplicates.js` and `diff-datasets.js` (broken before the repo was
|
||||
unified — a hardcoded `D:/` path in one, an undeclared `better-sqlite3` in the
|
||||
other, zero callers either way), plus `db-stats.js` and `verify-parity.js` as
|
||||
superseded by `differential-parity.mjs`.
|
||||
|
||||
The fidelity oracle is kept and marked **frozen** in its own header: produced by the
|
||||
Rust reader, so unregenerable, but still failing on any single-cell change across
|
||||
the 299 inputs. `regen-fidelity-hashes.sh` deleted, since it shelled out to cargo.
|
||||
|
||||
### Revert procedure
|
||||
|
||||
```bash
|
||||
git revert --no-commit 00a08d5 # restore parser/ and the removed scripts
|
||||
git checkout pre-go-parser-removal -- . # or take the whole pre-removal tree
|
||||
git checkout pre-go-parser-removal # or just inspect it
|
||||
```
|
||||
|
||||
The tag is the recovery point: at `0eb1747` both parsers exist and both test suites
|
||||
pass, so the differential gate can be re-run at any time.
|
||||
|
||||
## Unresolved
|
||||
|
||||
- Nothing blocking. The branch `refactor/go-parser` is unpushed; CI has therefore
|
||||
not run the new workflow yet, and the `pull_request` trigger it adds cannot be
|
||||
exercised until a PR exists.
|
||||
@@ -4,7 +4,7 @@ description: >-
|
||||
Build go-parser/ alongside the Rust parser/, validated by differential
|
||||
comparison against live Rust output. Rust stays working until parity is signed
|
||||
off.
|
||||
status: pending
|
||||
status: completed
|
||||
priority: P2
|
||||
branch: main
|
||||
tags:
|
||||
@@ -63,7 +63,7 @@ renders to, so they can now be re-derived without guessing.
|
||||
| 4 | [Reader writer and CLI](./phase-04-reader-writer-and-cli.md) | Completed |
|
||||
| 5 | [2016 format detection](./phase-05-2016-format-detection.md) | Completed |
|
||||
| 6 | [Differential parity gate](./phase-06-differential-parity-gate.md) | Completed |
|
||||
| 7 | [CI docs and cutover](./phase-07-ci-docs-and-script-port.md) | Pending |
|
||||
| 7 | [CI docs and cutover](./phase-07-ci-docs-and-script-port.md) | Completed |
|
||||
|
||||
Strictly sequential: 1 → 2 → 3 → 4 → 5 → 6 → 7. Phases 1 and 6 are hard gates.
|
||||
|
||||
@@ -95,17 +95,17 @@ data is untouched.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] For all 4 datasets, Go-built and Rust-built DBs are **logically equivalent**: identical
|
||||
- [x] For all 4 datasets, Go-built and Rust-built DBs are **logically equivalent**: identical
|
||||
row counts, identical per-column non-NULL counts across all 22 columns, identical
|
||||
`PRAGMA table_info`/`index_list`, and identical sorted full-table SHA-256
|
||||
- [ ] Both binaries emit identical build stdout per dataset, modulo the `Size:` line
|
||||
- [x] Both binaries emit identical build stdout per dataset, modulo the `Size:` line
|
||||
(this is the only check that covers the `source_rows`/`skipped` counters)
|
||||
- [ ] Go tests pass, including reader-fidelity tests against real files
|
||||
- [ ] `npm run build:db` produces four `.db.gz` via the Go binary, with a **row-count guard**
|
||||
- [x] Go tests pass, including reader-fidelity tests against real files
|
||||
- [x] `npm run build:db` produces four `.db.gz` via the Go binary, with a **row-count guard**
|
||||
that fails the build on deviation
|
||||
- [ ] CI green with Go toolchain, Rust actions removed, and a branch-verify path that does not
|
||||
- [x] CI green with Go toolchain, Rust actions removed, and a branch-verify path that does not
|
||||
publish to production
|
||||
- [ ] Frontend loads all 4 datasets unchanged, including accent-insensitive search
|
||||
- [x] Frontend loads all 4 datasets unchanged, including accent-insensitive search
|
||||
|
||||
**Explicitly not a criterion:** byte-identical databases. SQLite writes its own version number
|
||||
into header bytes 96-99, `VACUUM` rewrites page layout per-version, and `gzip -9` without `-n`
|
||||
@@ -152,7 +152,7 @@ Inputs:
|
||||
| 4 | No deploy guard — empty/truncated DB ships with green CI | Critical | Accept | Completed |
|
||||
| 5 | "Verify on a branch" unexecutable; `workflow_dispatch` publishes to prod | Critical | Accept | Completed |
|
||||
| 6 | Counter divergence invisible; 63 trailing empty sheets in `data/2017` | Critical | Accept | Completed |
|
||||
| 7 | Phase 7e breaks `npm run build:db`; 33 refs vs 8 listed | High | Accept | Phase 7 |
|
||||
| 7 | Phase 7e breaks `npm run build:db`; 33 refs vs 8 listed | High | Accept | Completed |
|
||||
| 8 | "byte-equivalent databases" provably unachievable | High | Accept | plan.md |
|
||||
| 9 | Phase 3 cites 20 of 29 tests; omitted 9 are the flagged traps | High | Accept | Phase 3 |
|
||||
| 10 | Two incompatible reader contracts; `[][]string` lossy | High | Accept | Phase 1, 4 |
|
||||
|
||||
Reference in new issue
Block a user