feat: download the database instead of reading it over HTTP

Reading the file where it lay never worked well enough. Two costs were
structural rather than bugs: reads are serial, because the worker uses
synchronous XHR, so a name search touching 390 pages waited 17 seconds
to move 608 KB — roughly one request per result row, which no page size
removes — and the first visitor after each deploy waited ~26 seconds for
the CDN to fill its cache with a 288 MB object.

The browser now downloads the whole database once and queries it in
memory with sql.js. A dataset page is gated behind that: the gate states
what it will cost, in transfer and in memory, and offers only the
download, because there is nothing to show without it.

Dropping the structures that existed to make range-request queries
index-driven halved the file. name_word carried one row per word of
every name, about 3.5 million of them, and with the partial score
indexes it was more than half of what every visitor would now download.
Measured on rebuilt databases: 2016 went 288.6 -> 142.5 MB (31 MB
gzipped on the wire), 2017 237.7 -> 119.3 MB, both with row counts and
audits unchanged. Queries on the result: an exam number is immediate, a
name scans all 877,460 rows in about 240 ms.

Alternatives were measured before choosing this. sqlite-wasm-http sizes
files from a HEAD Content-Length with no override, so on a host that
gzips it silently uses the compressed size. DuckDB-WASM ships 32-37 MB
of WebAssembly before its Parquet extension, more than this whole
download. Static pre-generated shards are the most robust option but
cannot answer arbitrary SQL, and cannot stop early the way LIMIT does.

The published name loses its chunk index, the byte budgets and the SQL
consent modal go with the range reads that made them necessary, and the
docs no longer describe a design the site does not use.
This commit is contained in:
tiennm99 committed 2026-08-14 17:43:10 +07:00
1 parent cbd262e5db
commit 4cb0a4f340
28 files changed
+449 -986

No files matched your search

+7 -10
View File
@@ -124,26 +124,23 @@ jobs:
- uses: actions/checkout@v7
# The site reads the databases a page at a time over range requests, so
# what matters is not that the host leaves the file alone in general — it
# gzips the un-ranged response, and browsers work around that by sending
# Accept-Encoding: identity whenever a request carries a Range header —
# but that a ranged read returns raw database bytes. Checking headers is
# what missed this before: a bare `curl -sI` advertises no encoding and
# so passes whatever the host does. Check the bytes instead.
- name: Verify ranged reads return database bytes
# Checks the published file is a database, not an error page or a
# truncated upload, by reading its first bytes. A range request is used
# only because it is the cheapest way to see them without pulling 142 MB;
# the site itself downloads the file whole.
- name: Verify the published databases are readable
env:
PAGE_URL: ${{ steps.deployment.outputs.page_url }}
run: |
set -euo pipefail
for id in $(jq -r '.datasets[].id' datasets.json); do
url="${PAGE_URL%/}/db/${id}.sqlite30"
url="${PAGE_URL%/}/db/${id}.sqlite3"
if ! magic=$(curl -sf -r 0-14 -H 'Accept-Encoding: identity;q=1, *;q=0' "$url"); then
echo "::error::$url is not fetchable"
exit 1
fi
if [ "$magic" != "SQLite format 3" ]; then
echo "::error::$url did not return database bytes over a range request"
echo "::error::$url does not start with the SQLite header"
exit 1
fi
echo "$url: SQLite format 3"