0 Fork 0

mirror of https://github.com/tiennm99/thptqg.git synced 2026-10-14 06:18:14 +00:00

Files

T

tiennm99 6ab7c1f855 fix(web): read the database in chunked mode so the length can be supplied

The previous attempt passed fileLength in the inline config, and the
worker discarded it. sqlite.worker.ts builds the lazy file's config
itself and hardcodes:

  fileLength: config.serverMode === "chunked"
    ? config.databaseLengthBytes
    : undefined

So in full mode there is no way to supply a length, and the library
falls back to sizing the file with a HEAD request — which GitHub Pages
answers with the gzipped length, and then refuses to use.

Chunked mode is the only mode that takes a length. One chunk holds the
whole database, so the chunk index is always 0 and every request goes to
urlPrefix + "0"; the assembler therefore publishes <id>.sqlite30. The
length still comes from the range probe, which also checks the bytes are
a SQLite header.

dbPrefixOf and dbOf derive one form from the other and are handed to
RemoteDatabase together, so the prefix the library appends an index to
and the file the assembler writes cannot drift apart. A test pins that;
nothing else would catch it, because the symptom is a 404 per query.

The stray-artifact guard now also rejects a leftover <id>.sqlite3, which
after this change is a stale artifact rather than the published one.

sqlite-wasm-http was checked as an alternative and does not help: its
worker takes the size from a HEAD Content-Length too, and its options
expose no way to override it, so on Pages it would silently use the
compressed size. Its shared-cache backend wants COOP/COEP, but it ships
a fallback that does not, so isolation was never the blocker — the
architecture note claiming otherwise is corrected.

2026-08-14 16:02:57 +07:00

4.7 KiB

Raw Blame History

thptqg

Exam-score lookup for Vietnam's national high school graduation exam. Four stages, each independent: crawler/ (Go) fetches spreadsheets into data/<id>/, parser/ (Go) turns them into <id>.db, assembler/ (Go) verifies, compresses and builds _site/, and web/ (SvelteKit + Tailwind) serves every dataset from one app.

datasets.json at the root is the registry the Go stages and the web app both read. See docs/ for architecture, pipeline and deployment.

Verifying a change

Run parser and assembler:

go -C parser test ./...
go -C assembler test ./...

Add the web checks when frontend files changed:

(cd web && npm test && npm run lint && npm run build)

npm run lint is ESLint. The web app is plain JavaScript — no TypeScript, no type-check step.

Do not run the crawler suite as part of routine verification. The crawler is not part of the build — data/<id>/ is committed and a crawl only refreshes it by hand — so its tests do not gate ordinary changes. Run them only when the change actually touches crawler/. CI still runs all three modules.

parser/internal/reader is the slow package (~9s) because the fidelity suite hashes every real input file. That is the point of it; do not skip it.

Things that look wrong but are not

data/2016/ filenames are content hashes and must stay verbatim. The parser sorts inputs bytewise and inserts last-wins, so filenames would decide which row survives a duplicate exam number. Renaming them also breaks parser/testdata/reader-fidelity-hashes.tsv, which is keyed by full path.
parser/testdata/reader-fidelity-hashes.tsv is frozen and cannot be regenerated. A mismatch is a reader bug until proven otherwise, never a cue to refresh the file. parser/cmd/dumpcells narrows a failure to the cell.
ToAscii filters the literal range U+0300..U+036F, not unicode.Mn. It must match toAscii in web/src/lib/to-ascii.js, or accent-insensitive search silently misses rows. to-ascii.test.js pins the pairs both sides must agree on.
The 2016 files use four different layouts, and detection is per sheet. Two of them publish scores in one column per subject instead of a DIEM_THI sentence, and one puts a three-row ministry title block above its header. parser/internal/ingest/detect2016.go holds the header tables; they are observations about 119 specific files, not a rule to generalise.
paths.relative is false in web/svelte.config.js. Every route is prerendered to its own file and asset URLs stay absolute, so the copy of index.html serving as 404.html works at any depth. No SPA 404-fallback is used — a fallback would break the ?q= deep links.
dbSizeMb in datasets.json is a build guard, not just a label. The assembler refuses to publish an artifact that falls below a ratio of it.
The databases ship uncompressed, as <id>.sqlite30. The trailing 0 is a chunk index, not a typo — see the next point. The browser reads byte ranges of the file, and a range of a gzip stream is not a range of the database.
The site uses chunked mode over a single chunk, and that is deliberate. GitHub Pages gzips application/octet-stream, so the HEAD request sql.js-httpvfs sizes a file with reports the compressed length, and the library refuses to open the file. Chunked mode is the only mode whose config takes a length (databaseLengthBytes); in full mode the worker hardcodes it to undefined, so a length passed there is silently dropped. One chunk holds the database, so the index is always 0 and every request goes to <id>.sqlite3 + 0. web/src/lib/db-probe.js supplies the length by reading the file header over a range request.
Ranged reads were never affected by the compression, because browsers must send Accept-Encoding: identity whenever a request carries a Range header. Verify the way a browser asks — curl -s -r 0-14 -H 'Accept-Encoding: identity' … must print SQLite format 3 — never a bare curl -sI, which advertises no encoding and so passes whatever the host does.
Every query the site runs must be index-driven. Over range requests an unindexed query fetches the whole table. Hence no index on ho_ten (nothing can use one), name_word for name search, partial indexes for the score presets, and the footer count read from datasets.json instead of COUNT(*).

Conventions

Comments state current behavior and why. History belongs in git and docs/, not in code comments.
Go files are CRLF in this checkout, so gofmt -l flags every file. It is not usable as a formatting gate as things stand.
Conventional commits, no AI references.

4.7 KiB Raw Blame History

thptqg

Verifying a change

Things that look wrong but are not

Conventions

4.7 KiB

Raw Blame History