docs: describe the design the code has, not the two before it

Three reversals had landed without the documentation following them, so the
docs described a pipeline that compresses its output, a schema with three
secondary indexes, and a browser that re-downloads the file on every visit.
None of those are true any more.

- Compression: the assembler stopped producing .gz when the databases began
  shipping as .sqlite3. The deployment guide's "why no uncompressed database
  can ship" section explained a guard that now exists for the opposite reason
  — to keep .db, .gz and journals out, so .sqlite3 stays the only name.
- Indexes: the architecture printed a DDL with three CREATE INDEX statements
  and a paragraph on the partial one. schema.go carries none.
- Persistence: "the download is repeated every visit ... has not been done"
  was listed as an open risk after db-cache.js closed it. Replaced with the
  ETag flow, the offline fallback, and the risks that did replace it.

Measured both transfers rather than scaling one from the other, which would
have been wrong: 2016 is 142 MB stored and 31 MB delivered, 2017 is 119 MB and
36 MB. The smaller database is the larger download, so neither figure follows
from the stored size.

Also corrects a CHUNK_BYTES reference to a module that no longer exists, the
238-289 MB per-dataset figure, two paths to web/src/lib/datasets.js, and the
CI step list, which omitted npm test and the post-deploy header check.

The two code comments that said the same outdated things go with them.
This commit is contained in:
tiennm99 committed 2026-08-15 17:35:09 +07:00
1 parent d8b5b66fcf
commit 2c24943cc6
11 files changed
+112 -52

No files matched your search

+34 -17
View File
@@ -7,13 +7,24 @@ One-time setup: **Settings → Pages → Source: GitHub Actions**.
## What the workflow does
1. Checkout, Go toolchain, Node 24, `npm ci` in `web/`
2. Parser, crawler and assembler test suites, web lint, `govulncheck` over all three modules
1. Checkout, Go toolchain (version taken from `parser/go.mod`), Node 24,
`npm ci` in `web/`
2. Parser, crawler and assembler test suites, `npm test` and `npm run lint` in
`web/`, `govulncheck` over all three Go modules
3. `go -C assembler run ./cmd/assemble` — the whole pipeline: compile the
parser, build and verify each database into `.build/public/db/`, run the web
build, assemble `_site/`. The databases are restored from the Actions cache
when nothing that determines them has changed, and only the site is built
4. `actions/upload-pages-artifact` + `actions/deploy-pages`
5. After deploying, each published database is read back from the live URL and
must begin `SQLite format 3` — proof the site serves a database rather than
an error page or a truncated upload
Pull requests run steps 1–3 and stop. The deploy job is guarded to `main`, so a
branch is verified end to end without touching the live site. The concurrency
group is keyed by ref rather than shared, because a shared lane let a
pull-request run cancel an in-flight `main` deploy while every check stayed
green.
The database build dominates the runtime: roughly 348 MB of Excel to parse. It
is cached in Actions, keyed on `data/**`, `parser/**` and `datasets.json`, so
@@ -62,39 +73,44 @@ shows up as a blank page with 404s on `/_app/...`.
indices, SBD validation, header tokens, blank-row stripping. No SQL: the
schema is canonical and lives in `parser/internal/schema/schema.go`
3. Add an entry to `datasets.json` — id, `expectedRows`, `dbSizeMb`
4. Add its presentation to `CONTENT` in `web/src/datasets.js`
4. Add its presentation to `CONTENT` in `web/src/lib/datasets.js`
Nothing else. The assembler and the router both read the registry, and the
frontend adapts to whichever columns the dataset populates. The last two steps
check each other, so forgetting either fails rather than half-working.
## Why no uncompressed database can ship
## What the databases are published as
The assembler deletes the source once compression succeeds, so the raw file
does not survive the build, and it then fails the job if any `.db`,
`.db-journal`, `.db-wal` or `.db-shm` reached the output.
One uncompressed `<id>.sqlite3` per dataset. Pages gzips it on the wire anyway,
so a `.gz` artifact would only mean decompressing twice.
Both guards exist because the previous pipeline wrote a 100+ MB uncompressed
database into the source tree and relied on an `rm` step to keep it out of the
artifact — one missing line away from publishing it.
`.sqlite3` is the only accepted name, and that is deliberate: it leaves `.db`
free, so site assembly can treat any stray `.db`, `.db.gz`, `.sqlite30` or
SQLite journal (`-journal`, `-wal`, `-shm`) in the output as the mistake it is
and fail the job. An earlier pipeline wrote a database into the source tree and
relied on an `rm` step to keep it out of the artifact — one missing line away
from publishing something it did not mean to.
## Notes
- **The database is not cacheable across deploys.** Every rebuild lays SQLite
pages out differently, so the file changes even when the data does not. Only
the pages a query touches are fetched, so this costs far less than it used
to, but a deploy does invalidate what a returning visitor had cached.
- **A deploy invalidates what returning visitors kept.** Every rebuild lays
SQLite pages out differently, so the file — and the ETag the browser stored it
under — changes even when the data does not. The visitor downloads once more
and keeps that copy until the next deploy.
- **The 100 MB file limit is a Git limit, not a Pages one.** It applies to
files committed to a repository; the databases are built in CI and uploaded
as a Pages artifact, and the documented Pages limits are a 1 GB published
site and 100 GB/month of bandwidth, with no per-file figure. The two
databases are 142 MB and 119 MB.
- **Total artifact is about 262 MB**, comfortably inside the 1 GB site limit.
- **Total artifact is about 263 MB**, comfortably inside the 1 GB site limit.
Dropping the search index the range-request design needed halved both files.
- **Pages compresses the databases on the wire, which is a benefit here.** The
extension is unknown to Pages, so the file is served as
`application/octet-stream`, which `mime-db` marks compressible: 142 MB stored
becomes about 31 MB delivered, and the browser decompresses it transparently.
`application/octet-stream`, which `mime-db` marks compressible, and the
browser decompresses it transparently: 142 MB stored becomes about 31 MB
delivered for 2016, and 119 MB becomes about 36 MB for 2017. Note that the
smaller database is the larger download: the two compress differently, so
neither transfer figure can be derived from the stored size.
- **Verify the bytes, not the headers.** A bare `curl -sI` reports success
whatever the host does. Read the file's first bytes instead:
@@ -119,3 +135,4 @@ run rebuilds the older state. There is no data to migrate.
| Tab crashes on a phone | The database needs 142 MB of memory for 2016, 119 MB for 2017; a low-memory device may have the tab killed |
| Deploy fails on assembly | A stray database artefact reached the output — a journal, a `.gz`, or a name from an earlier design; the error names the files |
| Missing rows after a data update | Unknown Excel header — check the per-file row counts the parser prints |
| Visitor still sees old data after a deploy | Their stored copy is keyed by ETag, so this should not happen. Confirm the deployed file's `ETag` header actually changed |