perf(parser): write the databases with 1 KiB pages

The browser fetches this file one page per HTTP request, so the page size
is the granularity of every read. At SQLite's 4 KiB default a row reached
by an index seek dragged 4 KB across the network; at 1 KiB it drags 1 KB.
A name search returns up to 100 scattered rows, so its row fetches fall
from about 400 KB to about 100 KB.

Measured on the rebuilt 2016 file: 6.3 rows share a page where 27 did.
The index walks are sequential and unaffected in bytes — the library's
read-ahead already collapses those into few requests.

Cost is 4% file size: 2016 288.6 -> 302.4 MB, 2017 237.7 -> 247.3 MB,
the site 528 -> 552 MB against the 1 GB GitHub Pages limit. Both
sql.js-httpvfs and sqlite-wasm-http recommend this page size.

The PRAGMA has to run before the DDL, since a page size is fixed once a
table exists, and requestChunkSize on the client has to match or every
page read spans two requests.

Row counts unchanged and through the assembler guards; query plans
re-checked and still index-driven on the rebuilt files.
This commit is contained in:
tiennm99 committed 2026-08-14 13:38:44 +07:00
1 parent dbc23c25c5
commit dbf13d0094
9 files changed
+280 -13

No files matched your search

+4
View File
@@ -181,6 +181,10 @@ silently drops 13,720 students** (Hanoi +7,275, HCM +6,445). That is what
## Expected row counts
The databases are written with 1 KiB pages (`PRAGMA page_size` in
`parser/internal/writer/writer.go`) because the browser fetches them a page per
HTTP request. `CHUNK_BYTES` in `web/src/lib/sqlite.svelte.ts` must match.
| id | Source rows | Skipped | DB rows |
| --- | --- | --- | --- |
| `2016` | 877,460 | 0 | **877,460** |
+2 -2
View File
@@ -83,8 +83,8 @@ artifact — one missing line away from publishing it.
files committed to a repository; the databases are built in CI and uploaded
as a Pages artifact, and the documented Pages limits are a 1 GB published
site and 100 GB/month of bandwidth, with no per-file figure. The two
databases are 289 MB and 238 MB.
- **Total artifact is about 528 MB**, inside the 1 GB site limit but with less
databases are 302 MB and 247 MB.
- **Total artifact is about 552 MB**, inside the 1 GB site limit but with less
headroom than before: a third dataset of this size would not fit. The fallback
is `sql.js-httpvfs`'s chunked mode, which splits a database into parts.
- **The server must not compress the databases.** Ranges of a compressed body
+2 -1
View File
@@ -175,6 +175,7 @@ total descending.
| WASM hosting | Bundled with the app | `sql.js-httpvfs` ships its own build; one less third-party runtime dependency |
| Diacritics search | Pre-computed `ho_ten_ascii`, indexed word by word | `LOWER(REPLACE(...))` at query time defeats the index, and `LIKE '%x%'` reads the whole table |
| Row count in the footer | Read from `datasets.json` | `COUNT(*)` scans an index — 20 MB over range requests |
| Page size | 1 KiB, matched by `requestChunkSize` | One HTTP request is one page; a row fetched by seek costs 1 KB rather than 4 KB, for about 5% more file |
| SQL safety | Leading-keyword allowlist | `sql.js` is in-memory so writes cannot persist; the allowlist prevents confusion |
| Row caps | 100 (lookup), 1000 (SQL) | Keeps DOM render sizes reasonable |
| Routing | SvelteKit file routes, prerendered | Each dataset gets a real HTML file with its own title |
@@ -191,7 +192,7 @@ total descending.
`curl -sI …/db/2016.sqlite3` must show no `content-encoding`.
- **`sql.js-httpvfs` is unmaintained** (0.8.12, September 2022) and ships its
own SQLite WASM. `sqlite-wasm-http`, on the official build, is the fallback.
- **Hosted size.** 528 MB for both datasets against the 1 GB GitHub Pages
- **Hosted size.** 552 MB for both datasets against the 1 GB GitHub Pages
limit; a third dataset of this size would not fit.
- **Excel format drift.** A new source file with an unseen header layout needs a
new branch in `parser/internal/ingest/detect2016.go` or a new config.