mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-10-20 02:18:27 +00:00
tiennm99
2c24943cc6
Three reversals had landed without the documentation following them, so the docs described a pipeline that compresses its output, a schema with three secondary indexes, and a browser that re-downloads the file on every visit. None of those are true any more. - Compression: the assembler stopped producing .gz when the databases began shipping as .sqlite3. The deployment guide's "why no uncompressed database can ship" section explained a guard that now exists for the opposite reason — to keep .db, .gz and journals out, so .sqlite3 stays the only name. - Indexes: the architecture printed a DDL with three CREATE INDEX statements and a paragraph on the partial one. schema.go carries none. - Persistence: "the download is repeated every visit ... has not been done" was listed as an open risk after db-cache.js closed it. Replaced with the ETag flow, the offline fallback, and the risks that did replace it. Measured both transfers rather than scaling one from the other, which would have been wrong: 2016 is 142 MB stored and 31 MB delivered, 2017 is 119 MB and 36 MB. The smaller database is the larger download, so neither figure follows from the stored size. Also corrects a CHUNK_BYTES reference to a module that no longer exists, the 238-289 MB per-dataset figure, two paths to web/src/lib/datasets.js, and the CI step list, which omitted npm test and the post-deploy header check. The two code comments that said the same outdated things go with them.
92 lines
4.7 KiB
Markdown
92 lines
4.7 KiB
Markdown
# thptqg
|
|
|
|
Exam-score lookup for Vietnam's national high school graduation exam. Four
|
|
stages, each independent: `crawler/` (Go) fetches spreadsheets into `data/<id>/`,
|
|
`parser/` (Go) turns them into `<id>.sqlite3`, `assembler/` (Go) verifies them
|
|
and builds `_site/`, and `web/` (SvelteKit + Tailwind) serves every dataset from
|
|
one app.
|
|
|
|
`datasets.json` at the root is the registry the Go stages and the web app both
|
|
read. See [`docs/`](./docs/) for architecture, pipeline and deployment.
|
|
|
|
## Verifying a change
|
|
|
|
Run parser and assembler:
|
|
|
|
```bash
|
|
go -C parser test ./...
|
|
go -C assembler test ./...
|
|
```
|
|
|
|
Add the web checks when frontend files changed:
|
|
|
|
```bash
|
|
(cd web && npm test && npm run lint && npm run build)
|
|
```
|
|
|
|
`npm run lint` is ESLint. The web app is plain JavaScript — no TypeScript, no
|
|
type-check step.
|
|
|
|
**Do not run the crawler suite as part of routine verification.** The crawler is
|
|
not part of the build — `data/<id>/` is committed and a crawl only refreshes it
|
|
by hand — so its tests do not gate ordinary changes. Run them only when the
|
|
change actually touches `crawler/`. CI still runs all three modules.
|
|
|
|
`parser/internal/reader` is the slow package (~9s) because the fidelity suite
|
|
hashes every real input file. That is the point of it; do not skip it.
|
|
|
|
## Things that look wrong but are not
|
|
|
|
- **`data/2016/` filenames are content hashes and must stay verbatim.** The
|
|
parser sorts inputs bytewise and inserts last-wins, so filenames would decide
|
|
which row survives a duplicate exam number. Renaming them also breaks
|
|
`parser/testdata/reader-fidelity-hashes.tsv`, which is keyed by full path.
|
|
- **`parser/testdata/reader-fidelity-hashes.tsv` is frozen and cannot be
|
|
regenerated.** A mismatch is a reader bug until proven otherwise, never a cue
|
|
to refresh the file. `parser/cmd/dumpcells` narrows a failure to the cell.
|
|
- **`ToAscii` filters the literal range U+0300..U+036F, not `unicode.Mn`.** It
|
|
must match `toAscii` in `web/src/lib/to-ascii.js`, or accent-insensitive search
|
|
silently misses rows. `to-ascii.test.js` pins the pairs both sides must agree
|
|
on.
|
|
- **The 2016 files use four different layouts, and detection is per sheet.**
|
|
Two of them publish scores in one column per subject instead of a `DIEM_THI`
|
|
sentence, and one puts a three-row ministry title block above its header.
|
|
`parser/internal/ingest/detect2016.go` holds the header tables; they are
|
|
observations about 119 specific files, not a rule to generalise.
|
|
- **`paths.relative` is false in `web/svelte.config.js`.** Every route is
|
|
prerendered to its own file and asset URLs stay absolute, so the copy of
|
|
`index.html` serving as `404.html` works at any depth. No SPA 404-fallback is
|
|
used — a fallback would break the `?q=` deep links.
|
|
- **`dbSizeMb` in `datasets.json` is a build guard and a user-facing figure.**
|
|
The assembler refuses to publish an artifact that falls below a ratio of it,
|
|
and the download gate shows it as the memory the tab will need.
|
|
- **The browser downloads the whole database and queries it in memory.** The
|
|
dataset page is gated behind that download: `download-gate.svelte` states the
|
|
transfer and memory cost and offers nothing but the download, because there
|
|
is no answer to give without it. `sql.js` holds the file in WebAssembly
|
|
memory for as long as the tab is open, so the memory figure is RAM, not disk.
|
|
- **The download is kept in Cache Storage, versioned by ETag** (`db-cache.js`).
|
|
A later visit opens the stored copy without asking, since consent was given
|
|
once and reuse costs no network; a redeploy changes the ETag, so the new
|
|
version replaces the old rather than being served stale. When the server
|
|
cannot be reached at all, any stored version is used — which is what lets the
|
|
site answer offline.
|
|
- **The schema carries no secondary indexes, deliberately.** An index saves a
|
|
scan that already takes a few hundred milliseconds in memory, and costs every
|
|
visitor megabytes of download. An earlier design read the file over HTTP
|
|
range requests and needed the opposite — a `name_word` table with one row per
|
|
word of every name, plus partial score indexes — which was more than half the
|
|
published file: 288 MB became 142 MB when they went.
|
|
- **The databases ship uncompressed, as `<id>.sqlite3`.** GitHub Pages gzips
|
|
them on the wire anyway, which is where the transfer figure comes from
|
|
(142 MB stored, 31 MB delivered), so publishing a `.gz` would only mean
|
|
decompressing twice.
|
|
|
|
## Conventions
|
|
|
|
- Comments state current behavior and why. History belongs in git and `docs/`,
|
|
not in code comments.
|
|
- Go files are CRLF in this checkout, so `gofmt -l` flags every file. It is not
|
|
usable as a formatting gate as things stand.
|
|
- Conventional commits, no AI references.
|