mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-10-13 22:16:35 +00:00
fix(web): take the database length from a range probe, not the host
GitHub Pages serves .sqlite3 as application/octet-stream, which mime-db marks compressible, so an un-ranged response comes back gzipped with the compressed length. sql.js-httpvfs sizes a file with a HEAD request, sees that length is unusable and refuses to open the database: Length of the file not known. It must either be supplied in the config or given by the HTTP server. Page reads were never affected. The Fetch standard requires browsers to send Accept-Encoding: identity on any request carrying a Range header, and the live site returns 206 with raw bytes to one. So the length is probed the same way and passed as fileLength, which is the escape hatch the library's own message points at. The probe reads the first 100 bytes, so it also checks the file starts with the SQLite magic and that its page size matches the request size — a host that ever compresses a ranged response now fails with a clear message rather than feeding the library the wrong bytes. The post-deploy check the docs prescribed could not have caught this: a bare `curl -sI` advertises no encoding, so it reports success whatever the host does. It is replaced, in the docs and in CI, by a ranged read that verifies the bytes.
This commit is contained in:
1 parent
3fed50cf8f
commit
9571fb7d55
7 files changed
+253
-12
No files matched your search
@@ -87,10 +87,23 @@ artifact — one missing line away from publishing it.
|
||||
- **Total artifact is about 552 MB**, inside the 1 GB site limit but with less
|
||||
headroom than before: a third dataset of this size would not fit. The fallback
|
||||
is `sql.js-httpvfs`'s chunked mode, which splits a database into parts.
|
||||
- **The server must not compress the databases.** Ranges of a compressed body
|
||||
address the wrong bytes, and the library refuses to open a file whose HEAD
|
||||
carries a `Content-Encoding`. `.sqlite3` is an unknown type to Pages, so it is
|
||||
served as `application/octet-stream` and left alone — verify after a deploy.
|
||||
- **Pages does compress the databases, and that is survivable.** `.sqlite3` is
|
||||
unknown to Pages, so it is served as `application/octet-stream`, which is
|
||||
marked compressible in `mime-db` and gzipped: a plain request returns
|
||||
`Content-Encoding: gzip` and the compressed length. Ranged reads are not
|
||||
affected, because the Fetch standard makes browsers send
|
||||
`Accept-Encoding: identity` on any request carrying a `Range` header. Only
|
||||
the length probe breaks, and the site supplies `fileLength` itself instead of
|
||||
trusting HEAD — see `web/src/lib/db-probe.js`.
|
||||
- **Verify the way a browser asks.** A bare `curl -sI` advertises no encoding
|
||||
and so reports success whatever the host does; it is what let this reach
|
||||
production. Check ranged reads instead, and check the bytes, not the headers:
|
||||
|
||||
```bash
|
||||
curl -s -r 0-15 -H 'Accept-Encoding: identity;q=1, *;q=0' \
|
||||
https://<user>.github.io/thptqg/db/2016.sqlite3 | head -c 16
|
||||
# must print: SQLite format 3
|
||||
```
|
||||
|
||||
## Rollback
|
||||
|
||||
@@ -104,7 +117,8 @@ run rebuilds the older state. There is no data to migrate.
|
||||
| Blank page, 404 on assets | `paths.base` in `svelte.config.js` does not match the repo name |
|
||||
| `Failed to fetch database: 404` | Dataset id in `datasets.json` does not match the file in `db/` |
|
||||
| A route 404s | The site step did not run, or the id is missing from `datasets.json` |
|
||||
| Database fails to open | The host compressed it. `curl -sI …/db/<id>.sqlite3` must show no `content-encoding`; ranges of a compressed body are unusable |
|
||||
| `Length of the file not known` | The host gzipped the un-ranged response, so HEAD reports the compressed size. The site supplies `fileLength` from a range probe; if this returns, that probe failed |
|
||||
| Database fails to open | A ranged read did not return raw database bytes. The range check above must print `SQLite format 3` |
|
||||
| Every query is slow or huge | It is not using an index. `EXPLAIN QUERY PLAN` it: a `SCAN` means the browser is fetching the whole table |
|
||||
| Deploy fails on assembly | An uncompressed database artefact reached the output; the error names the files |
|
||||
| Missing rows after a data update | Unknown Excel header — check the per-file row counts the parser prints |
|
||||
@@ -204,10 +204,15 @@ total descending.
|
||||
- **Unindexed queries are expensive.** The SQL tab can express a query that
|
||||
walks the table, which over range requests means fetching 100+ MB. A byte
|
||||
budget stops one before it gets that far, and the tab warns before it opens.
|
||||
- **`Content-Encoding` breaks everything.** If the host ever compresses
|
||||
`<id>.sqlite3` on the wire, ranges address compressed bytes and
|
||||
`sql.js-httpvfs` refuses to open the file. Verify after a deploy:
|
||||
`curl -sI …/db/2016.sqlite3` must show no `content-encoding`.
|
||||
- **`Content-Encoding` on a ranged response would break everything.** A range
|
||||
of a compressed body addresses the wrong bytes. In practice browsers prevent
|
||||
it: the Fetch standard requires `Accept-Encoding: identity` on any request
|
||||
carrying a `Range` header. GitHub Pages *does* gzip the un-ranged response —
|
||||
`application/octet-stream` is compressible in `mime-db` — which is why the
|
||||
file length is probed with a range request and passed as `fileLength` rather
|
||||
than left to the library's HEAD. `db-probe.js` checks the returned bytes
|
||||
start with the SQLite magic, so a host that ever compresses a ranged response
|
||||
fails loudly instead of returning nonsense.
|
||||
- **`sql.js-httpvfs` is unmaintained** (0.8.12, September 2022) and ships its
|
||||
own SQLite WASM. `sqlite-wasm-http`, on the official build, is the fallback.
|
||||
- **Hosted size.** 552 MB for both datasets against the 1 GB GitHub Pages
|
||||
|
||||
Reference in new issue
Block a user