mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-10-14 20:17:50 +00:00
docs: describe the design the code has, not the two before it
Three reversals had landed without the documentation following them, so the docs described a pipeline that compresses its output, a schema with three secondary indexes, and a browser that re-downloads the file on every visit. None of those are true any more. - Compression: the assembler stopped producing .gz when the databases began shipping as .sqlite3. The deployment guide's "why no uncompressed database can ship" section explained a guard that now exists for the opposite reason — to keep .db, .gz and journals out, so .sqlite3 stays the only name. - Indexes: the architecture printed a DDL with three CREATE INDEX statements and a paragraph on the partial one. schema.go carries none. - Persistence: "the download is repeated every visit ... has not been done" was listed as an open risk after db-cache.js closed it. Replaced with the ETag flow, the offline fallback, and the risks that did replace it. Measured both transfers rather than scaling one from the other, which would have been wrong: 2016 is 142 MB stored and 31 MB delivered, 2017 is 119 MB and 36 MB. The smaller database is the larger download, so neither figure follows from the stored size. Also corrects a CHUNK_BYTES reference to a module that no longer exists, the 238-289 MB per-dataset figure, two paths to web/src/lib/datasets.js, and the CI step list, which omitted npm test and the post-deploy header check. The two code comments that said the same outdated things go with them.
This commit is contained in:
1 parent
d8b5b66fcf
commit
2c24943cc6
11 files changed
+112
-52
No files matched your search
+34
-17
@@ -7,13 +7,24 @@ One-time setup: **Settings → Pages → Source: GitHub Actions**.
|
||||
|
||||
## What the workflow does
|
||||
|
||||
1. Checkout, Go toolchain, Node 24, `npm ci` in `web/`
|
||||
2. Parser, crawler and assembler test suites, web lint, `govulncheck` over all three modules
|
||||
1. Checkout, Go toolchain (version taken from `parser/go.mod`), Node 24,
|
||||
`npm ci` in `web/`
|
||||
2. Parser, crawler and assembler test suites, `npm test` and `npm run lint` in
|
||||
`web/`, `govulncheck` over all three Go modules
|
||||
3. `go -C assembler run ./cmd/assemble` — the whole pipeline: compile the
|
||||
parser, build and verify each database into `.build/public/db/`, run the web
|
||||
build, assemble `_site/`. The databases are restored from the Actions cache
|
||||
when nothing that determines them has changed, and only the site is built
|
||||
4. `actions/upload-pages-artifact` + `actions/deploy-pages`
|
||||
5. After deploying, each published database is read back from the live URL and
|
||||
must begin `SQLite format 3` — proof the site serves a database rather than
|
||||
an error page or a truncated upload
|
||||
|
||||
Pull requests run steps 1–3 and stop. The deploy job is guarded to `main`, so a
|
||||
branch is verified end to end without touching the live site. The concurrency
|
||||
group is keyed by ref rather than shared, because a shared lane let a
|
||||
pull-request run cancel an in-flight `main` deploy while every check stayed
|
||||
green.
|
||||
|
||||
The database build dominates the runtime: roughly 348 MB of Excel to parse. It
|
||||
is cached in Actions, keyed on `data/**`, `parser/**` and `datasets.json`, so
|
||||
@@ -62,39 +73,44 @@ shows up as a blank page with 404s on `/_app/...`.
|
||||
indices, SBD validation, header tokens, blank-row stripping. No SQL: the
|
||||
schema is canonical and lives in `parser/internal/schema/schema.go`
|
||||
3. Add an entry to `datasets.json` — id, `expectedRows`, `dbSizeMb`
|
||||
4. Add its presentation to `CONTENT` in `web/src/datasets.js`
|
||||
4. Add its presentation to `CONTENT` in `web/src/lib/datasets.js`
|
||||
|
||||
Nothing else. The assembler and the router both read the registry, and the
|
||||
frontend adapts to whichever columns the dataset populates. The last two steps
|
||||
check each other, so forgetting either fails rather than half-working.
|
||||
|
||||
## Why no uncompressed database can ship
|
||||
## What the databases are published as
|
||||
|
||||
The assembler deletes the source once compression succeeds, so the raw file
|
||||
does not survive the build, and it then fails the job if any `.db`,
|
||||
`.db-journal`, `.db-wal` or `.db-shm` reached the output.
|
||||
One uncompressed `<id>.sqlite3` per dataset. Pages gzips it on the wire anyway,
|
||||
so a `.gz` artifact would only mean decompressing twice.
|
||||
|
||||
Both guards exist because the previous pipeline wrote a 100+ MB uncompressed
|
||||
database into the source tree and relied on an `rm` step to keep it out of the
|
||||
artifact — one missing line away from publishing it.
|
||||
`.sqlite3` is the only accepted name, and that is deliberate: it leaves `.db`
|
||||
free, so site assembly can treat any stray `.db`, `.db.gz`, `.sqlite30` or
|
||||
SQLite journal (`-journal`, `-wal`, `-shm`) in the output as the mistake it is
|
||||
and fail the job. An earlier pipeline wrote a database into the source tree and
|
||||
relied on an `rm` step to keep it out of the artifact — one missing line away
|
||||
from publishing something it did not mean to.
|
||||
|
||||
## Notes
|
||||
|
||||
- **The database is not cacheable across deploys.** Every rebuild lays SQLite
|
||||
pages out differently, so the file changes even when the data does not. Only
|
||||
the pages a query touches are fetched, so this costs far less than it used
|
||||
to, but a deploy does invalidate what a returning visitor had cached.
|
||||
- **A deploy invalidates what returning visitors kept.** Every rebuild lays
|
||||
SQLite pages out differently, so the file — and the ETag the browser stored it
|
||||
under — changes even when the data does not. The visitor downloads once more
|
||||
and keeps that copy until the next deploy.
|
||||
- **The 100 MB file limit is a Git limit, not a Pages one.** It applies to
|
||||
files committed to a repository; the databases are built in CI and uploaded
|
||||
as a Pages artifact, and the documented Pages limits are a 1 GB published
|
||||
site and 100 GB/month of bandwidth, with no per-file figure. The two
|
||||
databases are 142 MB and 119 MB.
|
||||
- **Total artifact is about 262 MB**, comfortably inside the 1 GB site limit.
|
||||
- **Total artifact is about 263 MB**, comfortably inside the 1 GB site limit.
|
||||
Dropping the search index the range-request design needed halved both files.
|
||||
- **Pages compresses the databases on the wire, which is a benefit here.** The
|
||||
extension is unknown to Pages, so the file is served as
|
||||
`application/octet-stream`, which `mime-db` marks compressible: 142 MB stored
|
||||
becomes about 31 MB delivered, and the browser decompresses it transparently.
|
||||
`application/octet-stream`, which `mime-db` marks compressible, and the
|
||||
browser decompresses it transparently: 142 MB stored becomes about 31 MB
|
||||
delivered for 2016, and 119 MB becomes about 36 MB for 2017. Note that the
|
||||
smaller database is the larger download: the two compress differently, so
|
||||
neither transfer figure can be derived from the stored size.
|
||||
- **Verify the bytes, not the headers.** A bare `curl -sI` reports success
|
||||
whatever the host does. Read the file's first bytes instead:
|
||||
|
||||
@@ -119,3 +135,4 @@ run rebuilds the older state. There is no data to migrate.
|
||||
| Tab crashes on a phone | The database needs 142 MB of memory for 2016, 119 MB for 2017; a low-memory device may have the tab killed |
|
||||
| Deploy fails on assembly | A stray database artefact reached the output — a journal, a `.gz`, or a name from an earlier design; the error names the files |
|
||||
| Missing rows after a data update | Unknown Excel header — check the per-file row counts the parser prints |
|
||||
| Visitor still sees old data after a deploy | Their stored copy is keyed by ETag, so this should not happen. Confirm the deployed file's `ETag` header actually changed |
|
||||
Reference in new issue
Block a user