0 Fork 0

mirror of https://github.com/tiennm99/thptqg.git synced 2026-10-15 00:17:58 +00:00

Files

T

tiennm99 6ff2ed99ec refactor: collapse the two projects into one tree and move to npm

The repo held two near-duplicate projects. 2016/ and 2017/ each carried their
own React frontend, their own copy of the same Rust crate, and their own
package manager setup. 2016/tools/sync-from-thptqg2017.sh existed purely to
copy the parser source between them.

New layout:

  index.html + src/  the 2017 frontend, now the only one
  data/<id>/         2016, 2017, 2017-old, 2017-old2
  parser/            the single Rust crate, configs renamed to <id>.toml
  docs/              both projects' docs, 2016 copies suffixed -2016-legacy
                     pending the merge pass

<id> is now one identifier end to end: data/<id>/ feeds parser/configs/<id>.toml
and produces db/<id>.db.gz.

pnpm gives way to npm. pnpm-workspace.yaml existed only to whitelist
better-sqlite3's native build, which npm permits by default, so it has no
equivalent and is simply gone. Lockfiles cannot be converted; package-lock.json
is generated fresh. The migration direction is safe — pnpm's strict layout
forbids phantom dependencies, so anything that resolved under pnpm resolves
under npm's flat tree.

Adds parser/scripts/build-db.js and src/datasets.js: the four dataset IDs are
declared once and read by both the build tooling and (from the next phase) the
frontend.

Follow-on fixes the move made necessary:
  - eslint's Node-globals override pointed at scripts/, now parser/scripts/
  - crawl-baotintuc.js wrote to <root>/data, now data/2017
  - golden tests loaded configs by their old thptqg*-data.toml names

Drops the #[ignore]d Rust-vs-Node golden test. It shelled out to pnpm to run
scripts/build-database.js, a file removed when the parser was ported to Rust,
so it could never pass. check-duplicates.js and diff-datasets.js were already
broken before this change and are annotated as such rather than half-fixed.

63 Rust tests pass and clippy is clean from the new location.

2026-08-13 11:27:01 +07:00

5.2 KiB

Raw Blame History

System Architecture

Overview

A static serverless design: the entire dataset is packaged into a single SQLite file, gzip-compressed, and served as a static asset via GitHub Pages. The browser downloads it, decompresses it, and queries it in-process using sql.js (SQLite compiled to WebAssembly).

┌─────────────────┐   build      ┌──────────────────────┐
│  data/*.xlsx    │ ───────────▶ │ scripts/             │
│  (mixed formats)│              │  build-database.js   │
└─────────────────┘              │  (Node + xlsx +      │
                                 │   better-sqlite3)    │
                                 └──────────┬───────────┘
                                            │
                                            ▼
                                 ┌──────────────────────┐
                                 │ public/thptqg2016.db │
                                 └──────────┬───────────┘
                                            │ gzip -9 (CI)
                                            ▼
                                 ┌──────────────────────┐
                                 │ dist/thptqg2016.db.gz│
                                 │ dist/assets/*        │  ◀── Vite build
                                 └──────────┬───────────┘
                                            │ upload-pages-artifact
                                            ▼
                                    GitHub Pages CDN
                                            │
                                            ▼
                         ┌──────────────────────────────────┐
                         │ Browser                          │
                         │  ┌────────────────────────────┐  │
                         │  │ useSqlite hook             │  │
                         │  │  fetch(.db.gz) + stream    │  │
                         │  │  DecompressionStream gzip  │  │
                         │  │  sql.js WASM (from CDN)    │  │
                         │  └─────────────┬──────────────┘  │
                         │                ▼                  │
                         │  ┌────────────────────────────┐  │
                         │  │ React UI                   │  │
                         │  │  - SearchForm / ScoreTable │  │
                         │  │  - CustomQuery (SQL editor)│  │
                         │  └────────────────────────────┘  │
                         └──────────────────────────────────┘

Build-time data flow

Developer drops Excel files into data/.
npm run build:db reads every file; for each one:
- Detects the header row against a KNOWN_HEADERS set.
- Picks a format: separate-scores (one column per subject) vs. mapped (single DIEM_THI string).
- Parses each row into a canonical 18-column record.
- INSERT OR REPLACE into SQLite (primary key = so_bao_danh handles duplicates).
VACUUM shrinks the file.
CI runs gzip -k -9 → .db.gz.

Runtime flow

Page loads → React mounts → useSqlite("thptqg2016.db.gz").
Streaming fetch with a progress bar (driven by Content-Length).
DecompressionStream("gzip") decompresses on the fly.
sql.js loads its WASM from https://sql.js.org/dist/sql-wasm.wasm.
new SQL.Database(Uint8Array) — DB is now in RAM.
Each search / query → db.prepare() + stmt.step() loop → render.

Design decisions

Concern	Choice	Rationale
Storage	Static SQLite file	No backend needed; dataset is frozen
Compression	gzip in CI, `DecompressionStream` in browser	Native browser API; no extra library
WASM hosting	`sql.js.org` CDN	Smaller self-hosted artifact
Diacritics search	Pre-computed `ho_ten_ascii` column	`LOWER(REPLACE(...))` at query time defeats the index
SQL safety	Leading-keyword allowlist	`sql.js` is in-memory so writes don't persist, but the allowlist prevents user confusion
Row caps	100 (lookup), 1000 (SQL)	Keep DOM render sizes reasonable

Risks and limitations

DB size: tens of MB gzipped — slow links have a visible wait; mitigated by the progress bar.
Browser memory: the full DB lives in RAM; older mobile devices may OOM.
Dependency on sql.js.org: if that CDN is unreachable, WASM fails to load.
Excel format drift: a new source file with an unseen header layout needs a new branch in detectFormat().

5.2 KiB Raw Blame History