diff --git a/docs/data-pipeline.md b/docs/data-pipeline.md index 06c2681..82cb761 100644 --- a/docs/data-pipeline.md +++ b/docs/data-pipeline.md @@ -181,9 +181,10 @@ silently drops 13,720 students** (Hanoi +7,275, HCM +6,445). That is what ## Expected row counts -The databases are written with 1 KiB pages (`PRAGMA page_size` in +The databases are written with 4 KiB pages (`PRAGMA page_size` in `parser/internal/writer/writer.go`) because the browser fetches them a page per -HTTP request. `CHUNK_BYTES` in `web/src/lib/sqlite.svelte.js` must match. +HTTP request, and those requests are serial. `CHUNK_BYTES` in +`web/src/lib/sqlite.svelte.js` must match. | id | Source rows | Skipped | DB rows | | --- | --- | --- | --- | diff --git a/docs/system-architecture.md b/docs/system-architecture.md index 96cd913..7834311 100644 --- a/docs/system-architecture.md +++ b/docs/system-architecture.md @@ -176,7 +176,7 @@ total descending. | WASM hosting | Bundled with the app | `sql.js-httpvfs` ships its own build; one less third-party runtime dependency | | Diacritics search | Pre-computed `ho_ten_ascii`, indexed word by word | `LOWER(REPLACE(...))` at query time defeats the index, and `LIKE '%x%'` reads the whole table | | Row count in the footer | Read from `datasets.json` | `COUNT(*)` scans an index — 20 MB over range requests | -| Page size | 1 KiB, matched by `requestChunkSize` | One HTTP request is one page; a row fetched by seek costs 1 KB rather than 4 KB, for about 5% more file | +| Page size | 4 KiB, matched by `requestChunkSize` | One HTTP request is one page, and the worker issues them serially over synchronous XHR at ~40 ms each. Request count, not size, is what a search waits on: 390 requests moved 608 KB in 17 s. Four times the page is a quarter of the pages per scan. The 1 KiB both httpvfs libraries suggest optimises bytes per seek instead | | SQL safety | Leading-keyword allowlist | `sql.js` is in-memory so writes cannot persist; the allowlist prevents confusion | | Row caps | 100 (lookup), 1000 (SQL) | Keeps DOM render sizes reasonable | | Routing | SvelteKit file routes, prerendered | Each dataset gets a real HTML file with its own title | diff --git a/parser/internal/writer/writer.go b/parser/internal/writer/writer.go index 83ad96c..89230ac 100644 --- a/parser/internal/writer/writer.go +++ b/parser/internal/writer/writer.go @@ -46,14 +46,20 @@ func OpenDB(dbPath string) (*sql.DB, error) { // Before the DDL, because a page size cannot change once a table exists — // only the VACUUM in Finish could rewrite it, and only to this same value. // - // 1 KiB rather than SQLite's 4 KiB default because the browser reads this - // file a page at a time over HTTP: a row fetched by index seek costs one - // page, so a search that returns 100 scattered rows transfers 100 KB - // instead of 400 KB. It costs about 5% file size, and both sql.js-httpvfs - // and sqlite-wasm-http recommend it. web/src/lib/sqlite.svelte.js must - // request the same size, and web/src/lib/db-probe.js warns when a published - // database disagrees with it. - if _, err := db.Exec("PRAGMA page_size = 1024"); err != nil { + // SQLite's default, kept because the browser reads this file a page at a + // time over HTTP and what costs time is the number of requests, not their + // size. The library reads with synchronous XHR, so requests are strictly + // serial at about 40 ms each; a name search that made 390 of them took 17 + // seconds to move 608 KB. Four times the page size is a quarter of the + // pages for the same scan, and a shallower b-tree for each seek. + // + // The 1 KiB both sql.js-httpvfs and sqlite-wasm-http suggest optimises the + // other way — fewest bytes per seeked row — which is the wrong trade when + // one request costs more than the kilobyte it carries. + // + // web/src/lib/sqlite.svelte.js must request the same size, and + // web/src/lib/db-probe.js warns when a published database disagrees. + if _, err := db.Exec("PRAGMA page_size = 4096"); err != nil { db.Close() return nil, fmt.Errorf("set page size: %w", err) } diff --git a/web/src/lib/sqlite.svelte.js b/web/src/lib/sqlite.svelte.js index e6212c4..985c08f 100644 --- a/web/src/lib/sqlite.svelte.js +++ b/web/src/lib/sqlite.svelte.js @@ -18,7 +18,10 @@ import { probeDatabase } from "./db-probe.js"; // Must equal the page size the parser writes (PRAGMA page_size in // parser/internal/writer/writer.go), so one request is exactly one page. A // mismatch makes every logical page read span two requests. -const CHUNK_BYTES = 1024; +// +// Reads are serial — the worker uses synchronous XHR — so the count of +// requests, not their size, is what a search waits on. +const CHUNK_BYTES = 4096; /** Generous for indexed work: a name search costs well under 1 MB. */ export const SEARCH_BUDGET_BYTES = 25 * 1024 * 1024;