refactor: one directory per pipeline stage, and an assembler to drive them

The repository now reads as the pipeline it is: crawler fetches, parser
converts, assembler verifies and publishes, with data/ and web/ as the stores
they hand work through. go-parser is renamed parser now that there is no other.

The assembler replaces build-db.js and assemble-site.js. It compiles the
parser, builds and verifies each database, compresses it, runs the Vite build
and assembles _site — one command, and the only place that knows the order.

It also closes a real hole: nothing previously asserted that a database reached
the site. An empty staging directory assembled happily, so every page rendered,
every query 404d and CI stayed green. The row-count and size guards could not
catch that, since they only run when a database was built at all.

Removing Node from the root forced the dataset list out of web/src/datasets.js,
which the assembler cannot import. datasets.json is now the registry both sides
read — JSON because Go and the browser both parse it without a dependency —
while presentation stays in the web app, keyed by id and cross-checked against
the registry so a half-added dataset fails instead of half-working.

Guards verified by making each one fail: a missing database, and an expected
row count one higher than the truth.
This commit is contained in:
tiennm99 committed 2026-08-13 22:50:05 +07:00
1 parent b04f9844f9
commit c359a0b444
61 files changed
+1341 -480

No files matched your search

+27 -27
View File
@@ -23,9 +23,9 @@ jobs:
build:
runs-on: ubuntu-latest
env:
# The parser is pure Go — grate, excelize, yaml.v3 and modernc.org/sqlite
# are all cgo-free — so no C toolchain is needed. Set explicitly rather
# than relying on the default.
# Every Go module here is cgo-free — grate, excelize, yaml.v3, x/net,
# x/text and modernc.org/sqlite — so no C toolchain is needed. Set
# explicitly rather than relying on the default.
CGO_ENABLED: '0'
steps:
- uses: actions/checkout@v4
@@ -34,29 +34,37 @@ jobs:
with:
go-version: '1.26'
cache-dependency-path: |
go-parser/go.sum
parser/go.sum
crawler/go.sum
assembler/go.sum
# web/ is the only npm project in the repository; the other stages are Go.
- uses: actions/setup-node@v4
with:
node-version: '24'
cache: 'npm'
cache-dependency-path: package-lock.json
cache-dependency-path: web/package-lock.json
- run: npm ci
- name: Install web dependencies
working-directory: web
run: npm ci
# The reader-fidelity suite compares all 299 real input files against a
# The reader-fidelity suite compares every real input file against a
# committed hash oracle, so it is the regression guard for the whole
# reader. Runs before anything is built.
- name: Test parser
run: npm run test:go
run: go -C parser test ./...
# The crawler is not part of the build — it only refreshes data/ by hand.
# It is still compiled and tested here so it cannot rot unnoticed, and
# because its filename test guards the parser: input filenames decide
# which row survives a duplicate exam number.
# It is still tested here so it cannot rot unnoticed, and because its
# fixture test guards the parser: input filenames decide which row
# survives a duplicate exam number.
- name: Test crawler
run: npm run test:crawler
run: go -C crawler test ./...
- name: Lint web
working-directory: web
run: npm run lint
# excelize carries an open advisory, and the 2017 refresh runbook feeds
# network-downloaded spreadsheets straight into the parser.
@@ -64,22 +72,14 @@ jobs:
run: |
go install golang.org/x/vuln/cmd/govulncheck@latest
GOVULNCHECK="$(go env GOPATH)/bin/govulncheck"
(cd go-parser && "$GOVULNCHECK" ./...)
(cd crawler && "$GOVULNCHECK" ./...)
for m in parser crawler assembler; do (cd "$m" && "$GOVULNCHECK" ./...); done
# One parser binary builds every dataset; build-db.js reads the dataset
# list from web/src/datasets.js, verifies each database against its row
# count, then gzips it in place leaving no uncompressed file behind.
- name: Build databases
run: |
npm run build:go
npm run build:db
# One Vite build produces every page. web/scripts/assemble-site.js copies
# the emitted index.html to each dataset path (and the legacy nested
# URLs), then fails the job if an uncompressed database reached _site.
- name: Build and assemble site
run: npm run build:site
# One command runs the whole pipeline: compile the parser, build and
# verify each database against its registry row count, compress it, build
# the web app, and assemble _site — refusing to continue if a database is
# short, an artifact looks truncated, or one is missing entirely.
- name: Build site
run: go -C assembler run ./cmd/assemble
- uses: actions/upload-pages-artifact@v3
with: