mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-10-13 18:16:20 +00:00
perf(parser): write the databases with 1 KiB pages
The browser fetches this file one page per HTTP request, so the page size is the granularity of every read. At SQLite's 4 KiB default a row reached by an index seek dragged 4 KB across the network; at 1 KiB it drags 1 KB. A name search returns up to 100 scattered rows, so its row fetches fall from about 400 KB to about 100 KB. Measured on the rebuilt 2016 file: 6.3 rows share a page where 27 did. The index walks are sequential and unaffected in bytes — the library's read-ahead already collapses those into few requests. Cost is 4% file size: 2016 288.6 -> 302.4 MB, 2017 237.7 -> 247.3 MB, the site 528 -> 552 MB against the 1 GB GitHub Pages limit. Both sql.js-httpvfs and sqlite-wasm-http recommend this page size. The PRAGMA has to run before the DDL, since a page size is fixed once a table exists, and requestChunkSize on the client has to match or every page read spans two requests. Row counts unchanged and through the assembler guards; query plans re-checked and still index-driven on the rebuilt files.
This commit is contained in:
1 parent
dbc23c25c5
commit
dbf13d0094
9 files changed
+280
-13
No files matched your search
@@ -43,6 +43,19 @@ func OpenDB(dbPath string) (*sql.DB, error) {
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open db %s: %w", dbPath, err)
|
||||
}
|
||||
// Before the DDL, because a page size cannot change once a table exists —
|
||||
// only the VACUUM in Finish could rewrite it, and only to this same value.
|
||||
//
|
||||
// 1 KiB rather than SQLite's 4 KiB default because the browser reads this
|
||||
// file a page at a time over HTTP: a row fetched by index seek costs one
|
||||
// page, so a search that returns 100 scattered rows transfers 100 KB
|
||||
// instead of 400 KB. It costs about 5% file size, and both sql.js-httpvfs
|
||||
// and sqlite-wasm-http recommend it. web/src/lib/sqlite.svelte.ts must
|
||||
// request the same size.
|
||||
if _, err := db.Exec("PRAGMA page_size = 1024"); err != nil {
|
||||
db.Close()
|
||||
return nil, fmt.Errorf("set page size: %w", err)
|
||||
}
|
||||
if _, err := db.Exec(schema.DDL); err != nil {
|
||||
db.Close()
|
||||
return nil, fmt.Errorf("execute DDL: %w", err)
|
||||
|
||||
Reference in new issue
Block a user