0 Fork 0

mirror of https://github.com/tiennm99/thptqg.git synced 2026-10-13 00:15:14 +00:00

Files

T

tiennm99 c359a0b444 refactor: one directory per pipeline stage, and an assembler to drive them

The repository now reads as the pipeline it is: crawler fetches, parser
converts, assembler verifies and publishes, with data/ and web/ as the stores
they hand work through. go-parser is renamed parser now that there is no other.

The assembler replaces build-db.js and assemble-site.js. It compiles the
parser, builds and verifies each database, compresses it, runs the Vite build
and assembles _site — one command, and the only place that knows the order.

It also closes a real hole: nothing previously asserted that a database reached
the site. An empty staging directory assembled happily, so every page rendered,
every query 404d and CI stayed green. The row-count and size guards could not
catch that, since they only run when a database was built at all.

Removing Node from the root forced the dataset list out of web/src/datasets.js,
which the assembler cannot import. datasets.json is now the registry both sides
read — JSON because Go and the browser both parse it without a dependency —
while presentation stays in the web app, keyed by id and cross-checked against
the registry so a half-added dataset fails instead of half-working.

Guards verified by making each one fail: a missing database, and an expected
row count one higher than the truth.

2026-08-13 22:50:05 +07:00

4.2 KiB

Raw Blame History

Deployment Guide

Deploys to GitHub Pages via .github/workflows/deploy-pages.yml. Every push to main rebuilds both datasets and redeploys the whole site.

One-time setup: Settings → Pages → Source: GitHub Actions.

What the workflow does

Checkout, Go toolchain, Node 24, npm ci in web/
Parser and crawler test suites, web lint, govulncheck over all three modules
go -C assembler run ./cmd/assemble — the whole pipeline: compile the parser, build and verify each database, compress it into .build/public/db/, run the Vite build, assemble _site/
actions/upload-pages-artifact + actions/deploy-pages

The database build dominates the runtime: roughly 348 MB of Excel is parsed on every deploy.

Resulting URLs

https://<user>.github.io/thptqg/
https://<user>.github.io/thptqg/2016/
https://<user>.github.io/thptqg/2017/

/thptqg/2017/old/ and /thptqg/2017/old2/ were the pre-flattening URLs for the two removed 2017 archives. They are no longer served; like any unknown path they now render the hub via 404.html.

Local reproduction

(cd web && npm ci)
go -C assembler run ./cmd/assemble
npx serve _site

To rebuild a single dataset, or only the site:

go -C assembler run ./cmd/assemble db 2017
go -C assembler run ./cmd/assemble site

Base path

vite.config.js sets base: "/thptqg/". If you fork under a different repo name, update it to match — assets are referenced absolutely, so a mismatch shows up as a blank page with 404s on /assets/....

Adding a dataset

Put the Excel files in data/<id>/
Add parser/configs/<id>.yml with the parse rules — sheet mode, column indices, SBD validation, header tokens, blank-row stripping. No SQL: the schema is canonical and lives in parser/internal/schema/schema.go
Add an entry to datasets.json — id, expectedRows, dbSizeMb
Add its presentation to CONTENT in web/src/datasets.js

Nothing else. The assembler and the router both read the registry, and the frontend adapts to whichever columns the dataset populates. The last two steps check each other, so forgetting either fails rather than half-working.

Why no uncompressed database can ship

The assembler deletes the source once compression succeeds, so the raw file does not survive the build, and it then fails the job if any .db, .db-journal, .db-wal or .db-shm reached the output.

Both guards exist because the previous pipeline wrote a 100+ MB uncompressed database into the source tree and relied on an rm step to keep it out of the artifact — one missing line away from publishing it.

Notes

The gzipped database is not cacheable across deploys. Every rebuild produces a different .db.gz, because SQLite does not lay pages out deterministically. First-visit users pay the full download; later visits hit browser cache until the next deploy.
GitHub Pages caps individual files at 100 MB. The largest gzipped database is about 48 MB. Uncompressed they run 135–234 MB and would not fit — which is why the browser decompresses via DecompressionStream.
No server-side compression is assumed. The app fetches the .gz bytes directly rather than relying on Content-Encoding: gzip; Pages does not reliably compress arbitrary paths on the fly.
Total artifact is about 177 MB, well inside the 1 GB site limit.

Rollback

Deploys are stateless snapshots. Revert the commit on main and push; the next run rebuilds the older state. There is no data to migrate.

Troubleshooting

Symptom	Typical cause
Blank page, 404 on assets	`base` in `vite.config.js` does not match the repo name
`Failed to fetch database: 404`	Dataset id in `datasets.json` does not match the file in `db/`
A route 404s	The site step did not run, or the id is missing from `datasets.json`
WASM fails to load	`sql.js.org` unreachable — self-host `sql-wasm.wasm` and update `SQL_WASM_URL` in `use-sqlite.js`
Deploy fails on assembly	An uncompressed database artefact reached the output; the error names the files
Missing rows after a data update	Unknown Excel header — check the per-file row counts the parser prints

4.2 KiB Raw Blame History Unescape Escape