mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-10-14 00:17:03 +00:00
refactor: collapse the two projects into one tree and move to npm
The repo held two near-duplicate projects. 2016/ and 2017/ each carried their
own React frontend, their own copy of the same Rust crate, and their own
package manager setup. 2016/tools/sync-from-thptqg2017.sh existed purely to
copy the parser source between them.
New layout:
index.html + src/ the 2017 frontend, now the only one
data/<id>/ 2016, 2017, 2017-old, 2017-old2
parser/ the single Rust crate, configs renamed to <id>.toml
docs/ both projects' docs, 2016 copies suffixed -2016-legacy
pending the merge pass
<id> is now one identifier end to end: data/<id>/ feeds parser/configs/<id>.toml
and produces db/<id>.db.gz.
pnpm gives way to npm. pnpm-workspace.yaml existed only to whitelist
better-sqlite3's native build, which npm permits by default, so it has no
equivalent and is simply gone. Lockfiles cannot be converted; package-lock.json
is generated fresh. The migration direction is safe — pnpm's strict layout
forbids phantom dependencies, so anything that resolved under pnpm resolves
under npm's flat tree.
Adds parser/scripts/build-db.js and src/datasets.js: the four dataset IDs are
declared once and read by both the build tooling and (from the next phase) the
frontend.
Follow-on fixes the move made necessary:
- eslint's Node-globals override pointed at scripts/, now parser/scripts/
- crawl-baotintuc.js wrote to <root>/data, now data/2017
- golden tests loaded configs by their old thptqg*-data.toml names
Drops the #[ignore]d Rust-vs-Node golden test. It shelled out to pnpm to run
scripts/build-database.js, a file removed when the parser was ported to Rust,
so it could never pass. check-duplicates.js and diff-datasets.js were already
broken before this change and are annotated as such rather than half-fixed.
63 Rust tests pass and clippy is clean from the new location.
This commit is contained in:
1 parent
82db8295bb
commit
6ff2ed99ec
396 files changed
+2753
-8720
No files matched your search
@@ -0,0 +1,5 @@
|
||||
# Docs
|
||||
|
||||
- [`system-architecture.md`](./system-architecture.md) — data flow, schema, 3-variant deploy, score-tier model, admission-block catalog
|
||||
- [`data-pipeline.md`](./data-pipeline.md) — Excel parse quirks, overflow-sheet gotcha, audit scripts, refresh flow
|
||||
- [`deployment-guide.md`](./deployment-guide.md) — GitHub Pages workflow, adding a new variant, rollback
|
||||
@@ -0,0 +1,63 @@
|
||||
# Codebase Summary
|
||||
|
||||
## Directory layout
|
||||
|
||||
```
|
||||
thptqg2016/
|
||||
├── data/ # Source Excel files (~100, mixed formats)
|
||||
├── scripts/
|
||||
│ └── build-database.js # Parse Excel → SQLite (build-time, Node + better-sqlite3)
|
||||
├── public/
|
||||
│ └── thptqg2016.db # Generated DB, gzipped during CI
|
||||
├── src/
|
||||
│ ├── main.jsx # React entry
|
||||
│ ├── App.jsx # Root: tabs, lookup logic, useSqlite wiring
|
||||
│ ├── App.css / index.css # Design tokens, dark mode, a11y styles
|
||||
│ ├── hooks/
|
||||
│ │ └── use-sqlite.js # Fetch .db.gz + decompress + init sql.js
|
||||
│ └── components/
|
||||
│ ├── search-form.jsx # Input for exam ID / full name
|
||||
│ ├── score-table.jsx # Result table for lookups
|
||||
│ └── custom-query.jsx # SQL editor + presets + result grid
|
||||
├── .github/workflows/deploy.yml # CI: build db → gzip → vite build → Pages
|
||||
├── vite.config.js # base: "/thptqg2016/"
|
||||
└── eslint.config.js
|
||||
```
|
||||
|
||||
## Key modules
|
||||
|
||||
### `scripts/build-database.js`
|
||||
Build-time only. Reads every `.xlsx/.xls` in `data/`, detects the header format (three variants), parses the `DIEM_THI` string via regex or separate score columns, normalizes gender, derives a diacritics-stripped `ho_ten_ascii` column for accent-insensitive search, and inserts into SQLite with three indexes (`ho_ten`, `ho_ten_ascii`, `ten_cum_thi`).
|
||||
|
||||
### `src/hooks/use-sqlite.js`
|
||||
Streams `.db.gz` with download progress, decompresses via `DecompressionStream("gzip")`, loads `sql.js` (WASM served from the `sql.js.org` CDN), and returns `{ db, loading, error, progress }`.
|
||||
|
||||
### `src/App.jsx`
|
||||
Two tabs: **Lookup** and **Custom SQL**. Lookup auto-detects exam IDs (regex `^[A-Z]{2,4}\d+$`) vs names and picks one of three query paths: exact exam ID / ASCII LIKE / original + ASCII LIKE. Capped at 100 rows.
|
||||
|
||||
### `src/components/custom-query.jsx`
|
||||
Whitelists leading keywords (`SELECT`, `PRAGMA`, `EXPLAIN`, `WITH`), auto-appends `LIMIT 1000` when missing, measures `performance.now()` execution time, and ships 7 preset analytics queries.
|
||||
|
||||
## `student` table schema
|
||||
|
||||
```sql
|
||||
so_bao_danh TEXT PRIMARY KEY -- exam ID
|
||||
ho_ten TEXT NOT NULL -- full name
|
||||
ho_ten_ascii TEXT NOT NULL -- diacritics stripped, lowercased
|
||||
ngay_sinh TEXT -- date of birth
|
||||
ten_cum_thi TEXT -- exam cluster name
|
||||
gioi_tinh TEXT -- "Nam" | "Nữ" | NULL
|
||||
toan, ngu_van, vat_ly, hoa_hoc, -- REAL (nullable) subject scores
|
||||
sinh_hoc, lich_su, dia_ly,
|
||||
tieng_anh, tieng_phap, tieng_duc,
|
||||
tieng_nhat, tieng_trung
|
||||
```
|
||||
|
||||
Indexes: `idx_ho_ten`, `idx_ho_ten_ascii`, `idx_ten_cum_thi`.
|
||||
|
||||
## Conventions
|
||||
|
||||
- JS/JSX filenames: **kebab-case** (e.g., `search-form.jsx`, `use-sqlite.js`)
|
||||
- React components: `PascalCase` named exports
|
||||
- UI strings: Vietnamese (target audience)
|
||||
- Code comments: English; explain *why*, not *what*
|
||||
@@ -0,0 +1,110 @@
|
||||
# Data Pipeline
|
||||
|
||||
From raw Excel files to a compressed SQLite file the browser can load.
|
||||
|
||||
## Canonical source (data/)
|
||||
|
||||
63 Excel files from baotintuc.vn CDN. Source article:
|
||||
`https://baotintuc.vn/tuyen-sinh/tra-cuu-diem-thi-thpt-2017-cua-63-tinh-thanh-pho-tren-baotintucvn-20170706073512672.htm`
|
||||
|
||||
Re-download anytime:
|
||||
|
||||
```bash
|
||||
node scripts/crawl-baotintuc.js
|
||||
```
|
||||
|
||||
Idempotent — skips files already present. Saves to `data/<ascii-kebab-province>.xls`.
|
||||
|
||||
## Reference sources (data-old/, data-old2/)
|
||||
|
||||
Both were collected from Vietnamese news sites around the time of the 2017 exam.
|
||||
Specific publisher URLs were not recorded and cannot be recovered, so these
|
||||
datasets are **not reproducible** — the files committed in git are the only
|
||||
copy. They are kept for historical comparison and to expose the deploy matrix;
|
||||
the main deployment always uses the reproducible `data/` source.
|
||||
|
||||
## Source Excel shape
|
||||
|
||||
Every file is a single sheet (for `data-old/`) or up to 3 sheets (for `data/` .xls) with columns:
|
||||
|
||||
| Col | Name | Content |
|
||||
|---|---|---|
|
||||
| 0 | HO_TEN | full name in Vietnamese |
|
||||
| 1 | NGAY_SINH | `dd/mm/yyyy` |
|
||||
| 2 | SOBAODANH | 8-digit string, first 2 digits = province |
|
||||
| 3 | DIEM_THI | concatenated per-subject score string in the source language, e.g. `"Toan: 6.80 Van: 5.25 ... English: 5.80"` |
|
||||
|
||||
Some files lack a header row. `isHeaderRow()` in `build-lib.js` detects and skips.
|
||||
|
||||
## Score text parsing
|
||||
|
||||
`SCORE_PATTERNS` in `build-lib.js` defines one regex per subject. The patterns must match the exact subject labels the source files use (Vietnamese).
|
||||
|
||||
Subjects supported (DB column / label in source text):
|
||||
|
||||
| Column | Source label | English |
|
||||
|---|---|---|
|
||||
| toan | Toán | Math |
|
||||
| ngu_van | Ngữ văn | Literature |
|
||||
| vat_ly | Vật lí | Physics |
|
||||
| hoa_hoc | Hóa học | Chemistry |
|
||||
| sinh_hoc | Sinh học | Biology |
|
||||
| khtn | KHTN | Natural Sciences combined |
|
||||
| lich_su | Lịch sử | History |
|
||||
| dia_ly | Địa lí | Geography |
|
||||
| gdcd | GDCD | Civic Education |
|
||||
| khxh | KHXH | Social Sciences combined |
|
||||
| tieng_anh | Tiếng Anh | English |
|
||||
| tieng_phap | Tiếng Pháp | French |
|
||||
| tieng_nga | Tiếng Nga | Russian |
|
||||
| tieng_trung | Tiếng Trung | Chinese |
|
||||
|
||||
Missing-in-string → `NULL` in DB (student didn't take that subject).
|
||||
|
||||
## Why three parsers
|
||||
|
||||
One shared lib + three thin parsers, not one parser with flags. Each dataset's quirks stay visible in its own file:
|
||||
|
||||
| File | Source dir | Sheet strategy | Extra guard |
|
||||
|---|---|---|---|
|
||||
| `build-database.js` | `data/` | iterate ALL sheets (.xls row overflow) | — |
|
||||
| `build-database-old.js` | `data-old/` | sheet 0 only | reject rows where `so_bao_danh` isn't digits-only (rejects an `HO_TEN` → `SOBAODANH` header leak present in this export) |
|
||||
| `build-database-old2.js` | `data-old2/` | iterate ALL sheets (HCM overflow) | skip fully blank rows before counting |
|
||||
|
||||
## Overflow-sheet gotcha
|
||||
|
||||
The old `.xls` binary format caps at 65,536 rows per sheet. Hanoi and Ho Chi Minh City exceed that in the baotintuc export, so Excel auto-splits the overflow into `Sheet2`. **Only reading `Sheet1` silently drops 13,720 students** (Hanoi +7,275, HCM +6,445).
|
||||
|
||||
The audit `node scripts/audit-row-counts.js` catches this by comparing source-row-count (across all sheets) to DB-row-count.
|
||||
|
||||
## Audits
|
||||
|
||||
Run any time after a rebuild:
|
||||
|
||||
```bash
|
||||
node scripts/audit-row-counts.js # source rows vs DB rows (zero-loss check)
|
||||
node scripts/check-duplicates.js # md5 + row-content duplicates in data/
|
||||
node scripts/diff-datasets.js # compare two DBs (needs backup/ dir)
|
||||
```
|
||||
|
||||
Expected baseline:
|
||||
|
||||
| DB | Source rows | Skipped | DB rows |
|
||||
|---|---|---|---|
|
||||
| main | 861,131 | 63 empty rows | **861,068** |
|
||||
| old | 847,349 | 1 header-leak | **847,348** |
|
||||
| old2 | 679,764 | 0 | **679,764** |
|
||||
|
||||
## Refreshing the data
|
||||
|
||||
```bash
|
||||
rm data/*.xls # clear current snapshot
|
||||
node scripts/crawl-baotintuc.js # re-download from CDN
|
||||
npm run build:db # rebuild
|
||||
node scripts/audit-row-counts.js # verify
|
||||
gzip -kf -9 public/thptqg2017.db # compress for shipping
|
||||
```
|
||||
|
||||
## Adding a new source dataset
|
||||
|
||||
See `docs/deployment-guide.md` → "Adding a new variant".
|
||||
@@ -0,0 +1,50 @@
|
||||
# Deployment Guide
|
||||
|
||||
## Automatic (recommended)
|
||||
|
||||
Push to `main` → GitHub Actions builds and deploys to GitHub Pages automatically.
|
||||
|
||||
Workflow: `.github/workflows/deploy.yml`
|
||||
|
||||
CI steps:
|
||||
1. `npm ci`
|
||||
2. `npm run build:db` — generate `public/thptqg2016.db` from `data/*.xlsx`
|
||||
3. `gzip -k -9 public/thptqg2016.db` — max compression, keep original
|
||||
4. `npm run build` — Vite bundles `dist/`
|
||||
5. `rm -f dist/thptqg2016.db` — ship only the gzipped copy
|
||||
6. `actions/upload-pages-artifact@v3` + `actions/deploy-pages@v4`
|
||||
|
||||
One-time setup: **Settings → Pages → Source: GitHub Actions**.
|
||||
|
||||
## Manual (local verification)
|
||||
|
||||
```bash
|
||||
npm install
|
||||
npm run build:db
|
||||
gzip -k -9 public/thptqg2016.db # Linux/macOS; on Windows use 7zip or WSL
|
||||
npm run build
|
||||
rm dist/thptqg2016.db # optional, shrinks artifact
|
||||
npm run preview # serve dist/ locally
|
||||
```
|
||||
|
||||
Open <http://localhost:4173/thptqg2016/>.
|
||||
|
||||
## Base path
|
||||
|
||||
`vite.config.js` sets `base: "/thptqg2016/"`. If you fork under a different repo name, update this to match `<repo-name>` so assets resolve correctly on GitHub Pages.
|
||||
|
||||
## Updating data
|
||||
|
||||
1. Add the new Excel file to `data/`.
|
||||
2. If its header is unfamiliar, open `scripts/build-database.js` and extend `detectFormat()` or the `processSeparateScoresRow` / `processMappedRow` helpers.
|
||||
3. Run `npm run build:db` locally to check row counts and error skips.
|
||||
4. Commit + push → CI redeploys.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Typical cause |
|
||||
|---------|---------------|
|
||||
| Blank page, 404 on assets | `base` in `vite.config.js` doesn't match the repo name |
|
||||
| `Failed to fetch database: 404` | gzip step skipped, or `.db.gz` removed from `dist/` |
|
||||
| WASM fails to load | `sql.js.org` blocked / offline — self-host `sql-wasm.wasm` in `public/` and update `SQL_WASM_URL` in `use-sqlite.js` |
|
||||
| Missing rows after build | Excel file has an unknown header — check console for `Failed to read` or `errorCount` |
|
||||
@@ -0,0 +1,52 @@
|
||||
# Deployment Guide
|
||||
|
||||
Deploys to GitHub Pages via `.github/workflows/deploy.yml`. Every push to `main` rebuilds and redeploys all 3 variants.
|
||||
|
||||
## What the workflow does
|
||||
|
||||
1. Checkout, setup Node 20, `npm ci`
|
||||
2. `npm run build:db:all` — builds 3 SQLite DBs (main, old, old2)
|
||||
3. Gzip each DB at level 9
|
||||
4. `npm run build:all` — builds 3 Vite bundles into `dist/`, `dist/old/`, `dist/old2/`
|
||||
5. Remove uncompressed `.db` files from `dist/` (only `.gz` ships)
|
||||
6. `actions/upload-pages-artifact` + `actions/deploy-pages`
|
||||
|
||||
Total CI time ≈ 4–6 min (DB build dominates).
|
||||
|
||||
## Resulting URLs
|
||||
|
||||
- `https://<user>.github.io/thptqg2017/`
|
||||
- `https://<user>.github.io/thptqg2017/old/`
|
||||
- `https://<user>.github.io/thptqg2017/old2/`
|
||||
|
||||
## Local reproduction
|
||||
|
||||
```bash
|
||||
npm ci
|
||||
npm run build:db:all
|
||||
gzip -kf -9 public/thptqg2017.db public-old/thptqg2017.db public-old2/thptqg2017.db
|
||||
npm run build:all
|
||||
npx serve dist # or any static server
|
||||
```
|
||||
|
||||
## Adding a new variant
|
||||
|
||||
1. Drop source Excel files into a new `data-vN/` folder
|
||||
2. Add `public-vN/` to `.gitignore` patterns for the `.db` + `.db.gz`
|
||||
3. Copy `scripts/build-database-old.js` → `scripts/build-database-vN.js`; update `SRC_DIR` / `DB_PATH`; adjust parse logic for the new quirks (multi-sheet? blank rows? numeric guard?)
|
||||
4. Add variant to `VARIANT_CONFIG` in `vite.config.js`:
|
||||
```js
|
||||
vN: { base: "/thptqg2017/vN/", publicDir: "public-vN", outDir: "dist/vN" }
|
||||
```
|
||||
5. Add `build:db:vN` and `build:vN` npm scripts; wire them into `build:db:all` and `build:all`
|
||||
6. Update `.github/workflows/deploy.yml` — add gzip step + rm step for the new DB
|
||||
|
||||
## Notes
|
||||
|
||||
- **DB gzip is non-cacheable across deploys** — every rebuild produces a new `.db.gz` (not byte-identical due to SQLite page shuffling). First-visit users pay the 47 MB download; subsequent visits hit browser cache until the next deploy.
|
||||
- **GitHub Pages file size limit**: individual files ≤ 100 MB. Main gzipped DB is ~47 MB — safe. Uncompressed 159 MB DB would not fit; that's why we ship `.gz` and the frontend decompresses via `DecompressionStream`.
|
||||
- **No server-side compression assumption.** The app reads the `.gz` file directly (not `Content-Encoding: gzip`). GH Pages does not reliably gzip on-the-fly for arbitrary paths; shipping pre-gzipped bytes is deterministic.
|
||||
|
||||
## Rollback
|
||||
|
||||
Deploys are stateless snapshots. To rollback, revert the commit on `main` and push — the next workflow rebuilds the older state. There's no state to migrate.
|
||||
@@ -0,0 +1,34 @@
|
||||
# Project Overview — thptqg2016
|
||||
|
||||
## Goal
|
||||
|
||||
Provide a public lookup tool for Vietnam's 2016 National High School Graduation Exam scores (877,461 candidates), running entirely on the client, hosted for free on GitHub Pages.
|
||||
|
||||
## Scope
|
||||
|
||||
- Lookup by exam ID or full name (with Vietnamese diacritics handling)
|
||||
- Read-only SQL queries against a single `student` table
|
||||
- Static dataset — no updates (the 2016 exam is long over)
|
||||
|
||||
## Target users
|
||||
|
||||
- Former 2016 candidates checking their scores
|
||||
- Education researchers / data journalists running aggregate stats
|
||||
- Developers exploring SQL on a real-world dataset
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Zero backend**: the full DB (tens of MB gzipped) is downloaded to the browser
|
||||
- **Read-only**: INSERT/UPDATE/DELETE rejected to avoid the illusion that user edits persist
|
||||
- **Row caps**: 100 rows (lookup), 1000 rows (SQL) to prevent browser hangs
|
||||
- **Vietnamese-first UI**: app labels and data are Vietnamese; documentation is English
|
||||
|
||||
## Data sources
|
||||
|
||||
Excel files (`.xlsx`/`.xls`) collected from newspapers and exam clusters in 2016, stored in `data/`. One file per cluster, with several different column layouts (see `scripts/build-database.js`).
|
||||
|
||||
Original aggregator link: <https://dtntbacgiang.edu.vn/tin-tuc/tin-tuc-su-kien/cong-bo-diem-thi-thptqg-2016-toan-bo-120-cum-thi-da-co-diem.html> — no longer accessible, so the raw Excel files are mirrored in this repo under `data/`.
|
||||
|
||||
## Status
|
||||
|
||||
Stable. Data is frozen. Recent work focuses on UX polish (dark mode, accessibility, diacritics-insensitive search).
|
||||
@@ -0,0 +1,83 @@
|
||||
# System Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
A **static serverless** design: the entire dataset is packaged into a single SQLite file, gzip-compressed, and served as a static asset via GitHub Pages. The browser downloads it, decompresses it, and queries it in-process using `sql.js` (SQLite compiled to WebAssembly).
|
||||
|
||||
```
|
||||
┌─────────────────┐ build ┌──────────────────────┐
|
||||
│ data/*.xlsx │ ───────────▶ │ scripts/ │
|
||||
│ (mixed formats)│ │ build-database.js │
|
||||
└─────────────────┘ │ (Node + xlsx + │
|
||||
│ better-sqlite3) │
|
||||
└──────────┬───────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ public/thptqg2016.db │
|
||||
└──────────┬───────────┘
|
||||
│ gzip -9 (CI)
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ dist/thptqg2016.db.gz│
|
||||
│ dist/assets/* │ ◀── Vite build
|
||||
└──────────┬───────────┘
|
||||
│ upload-pages-artifact
|
||||
▼
|
||||
GitHub Pages CDN
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ Browser │
|
||||
│ ┌────────────────────────────┐ │
|
||||
│ │ useSqlite hook │ │
|
||||
│ │ fetch(.db.gz) + stream │ │
|
||||
│ │ DecompressionStream gzip │ │
|
||||
│ │ sql.js WASM (from CDN) │ │
|
||||
│ └─────────────┬──────────────┘ │
|
||||
│ ▼ │
|
||||
│ ┌────────────────────────────┐ │
|
||||
│ │ React UI │ │
|
||||
│ │ - SearchForm / ScoreTable │ │
|
||||
│ │ - CustomQuery (SQL editor)│ │
|
||||
│ └────────────────────────────┘ │
|
||||
└──────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Build-time data flow
|
||||
|
||||
1. Developer drops Excel files into `data/`.
|
||||
2. `npm run build:db` reads every file; for each one:
|
||||
- Detects the header row against a `KNOWN_HEADERS` set.
|
||||
- Picks a format: `separate-scores` (one column per subject) vs. `mapped` (single `DIEM_THI` string).
|
||||
- Parses each row into a canonical 18-column record.
|
||||
- `INSERT OR REPLACE` into SQLite (primary key = `so_bao_danh` handles duplicates).
|
||||
3. `VACUUM` shrinks the file.
|
||||
4. CI runs `gzip -k -9` → `.db.gz`.
|
||||
|
||||
## Runtime flow
|
||||
|
||||
1. Page loads → React mounts → `useSqlite("thptqg2016.db.gz")`.
|
||||
2. Streaming fetch with a progress bar (driven by `Content-Length`).
|
||||
3. `DecompressionStream("gzip")` decompresses on the fly.
|
||||
4. `sql.js` loads its WASM from `https://sql.js.org/dist/sql-wasm.wasm`.
|
||||
5. `new SQL.Database(Uint8Array)` — DB is now in RAM.
|
||||
6. Each search / query → `db.prepare()` + `stmt.step()` loop → render.
|
||||
|
||||
## Design decisions
|
||||
|
||||
| Concern | Choice | Rationale |
|
||||
|---------|--------|-----------|
|
||||
| Storage | Static SQLite file | No backend needed; dataset is frozen |
|
||||
| Compression | gzip in CI, `DecompressionStream` in browser | Native browser API; no extra library |
|
||||
| WASM hosting | `sql.js.org` CDN | Smaller self-hosted artifact |
|
||||
| Diacritics search | Pre-computed `ho_ten_ascii` column | `LOWER(REPLACE(...))` at query time defeats the index |
|
||||
| SQL safety | Leading-keyword allowlist | `sql.js` is in-memory so writes don't persist, but the allowlist prevents user confusion |
|
||||
| Row caps | 100 (lookup), 1000 (SQL) | Keep DOM render sizes reasonable |
|
||||
|
||||
## Risks and limitations
|
||||
|
||||
- **DB size**: tens of MB gzipped — slow links have a visible wait; mitigated by the progress bar.
|
||||
- **Browser memory**: the full DB lives in RAM; older mobile devices may OOM.
|
||||
- **Dependency on `sql.js.org`**: if that CDN is unreachable, WASM fails to load.
|
||||
- **Excel format drift**: a new source file with an unseen header layout needs a new branch in `detectFormat()`.
|
||||
@@ -0,0 +1,97 @@
|
||||
# System Architecture
|
||||
|
||||
Static site. No backend. Browser downloads a compressed SQLite file at boot, then every query runs locally via `sql.js` (WASM).
|
||||
|
||||
## Data flow
|
||||
|
||||
```
|
||||
Excel files (.xls / .xlsx)
|
||||
│
|
||||
▼ build-database*.js (Node)
|
||||
SQLite DB (public*/thptqg2017.db)
|
||||
│
|
||||
▼ gzip -9
|
||||
thptqg2017.db.gz (~47 MB for main variant)
|
||||
│
|
||||
▼ Vite build copies public*/ into dist/
|
||||
Static site on GitHub Pages
|
||||
│
|
||||
▼ browser loads
|
||||
sql.js (WASM) opens the .db → queries run client-side
|
||||
```
|
||||
|
||||
## Three deployment variants
|
||||
|
||||
One repo → three independent sites, same frontend, different dataset.
|
||||
|
||||
| Variant | Route | Source dir | Public dir | Build cmd |
|
||||
|---|---|---|---|---|
|
||||
| main | `/thptqg2017/` | `data/` | `public/` | `npm run build` |
|
||||
| old | `/thptqg2017/old/` | `data-old/` | `public-old/` | `npm run build:old` |
|
||||
| old2 | `/thptqg2017/old2/` | `data-old2/` | `public-old2/` | `npm run build:old2` |
|
||||
|
||||
`vite.config.js` reads `process.env.VARIANT` and switches `base`, `publicDir`, `outDir` accordingly. `emptyOutDir` is on only for the main build so the variant builds merge into `dist/old/`, `dist/old2/` cleanly.
|
||||
|
||||
## Schema (all 3 DBs share this)
|
||||
|
||||
```sql
|
||||
CREATE TABLE student (
|
||||
so_bao_danh TEXT PRIMARY KEY, -- exam ID (8 digits, first 2 = province)
|
||||
ho_ten TEXT NOT NULL,
|
||||
ho_ten_ascii TEXT NOT NULL, -- NFD-stripped lowercase for accent-insensitive search
|
||||
ngay_sinh TEXT, -- dd/mm/yyyy
|
||||
toan, ngu_van, vat_ly, hoa_hoc, sinh_hoc, khtn,
|
||||
lich_su, dia_ly, gdcd, khxh,
|
||||
tieng_anh, tieng_phap, tieng_nga, tieng_trung REAL
|
||||
);
|
||||
CREATE INDEX idx_ho_ten ON student(ho_ten);
|
||||
CREATE INDEX idx_ho_ten_ascii ON student(ho_ten_ascii);
|
||||
```
|
||||
|
||||
No German / Japanese language columns — neither appears in any source file.
|
||||
|
||||
Each score column is `NULL` when the student didn't take that subject. A student's "admission block" total is only computed when all 3 required subjects are non-null.
|
||||
|
||||
## Parse quirks (why 3 builder scripts)
|
||||
|
||||
Each dataset had a different quirk. Rather than one mega-parser with mode flags, each builder script is ~70 lines and isolates its own workarounds.
|
||||
|
||||
| Dataset | Quirk | Mitigation |
|
||||
|---|---|---|
|
||||
| data/ (baotintuc .xls) | Hanoi + HCM overflow past the 65,536 row-per-sheet .xls limit | Iterate all `wb.SheetNames` |
|
||||
| data-old/ (xlsx) | Single-sheet always; one bogus header row (`SOBAODANH`/`HO_TEN`) leaked into an earlier DB | Strict numeric-SBD guard rejects the leak |
|
||||
| data-old2/ (xlsx) | HCM overflow + many blank trailing rows | Multi-sheet walk + blank-row pre-filter |
|
||||
|
||||
Shared concerns (regex, schema, `toAscii`, header detection) live in `scripts/build-lib.js`.
|
||||
|
||||
## Admission blocks
|
||||
|
||||
Vietnamese universities admit students based on 3-subject combinations called "admission blocks" (khối thi). `src/lib/admission-blocks.js` lists all 49 official 2017 blocks computable from our schema (A00–A11, B00–B08, C00–C20, D01–D15). Each entry is `{ code, subjects: [3 keys], label }`.
|
||||
|
||||
`computeBlocks(student)` returns only blocks where the student has all 3 subject scores, sorted by total desc. Used by the student detail card.
|
||||
|
||||
The SQL preset "Top 10 best-block in Long An" materialises all 49 blocks as a `UNION ALL` CTE, picks each student's best block via `ROW_NUMBER() OVER (PARTITION BY so_bao_danh ORDER BY s DESC, k)`, then ranks across students.
|
||||
|
||||
## Score tiers (UI coding)
|
||||
|
||||
6-level TFT rarity ladder (white → green → blue → purple → gold → prismatic). Defined in `scoreTier()` in `src/lib/admission-blocks.js`.
|
||||
|
||||
| Tier | Range | UI label (Vietnamese) | English |
|
||||
|---|---|---|---|
|
||||
| common | ≤ 1 | Điểm liệt | Disqualifying |
|
||||
| uncommon | < 5 | Chưa đạt | Below passing |
|
||||
| rare | 5–6.5 | Trung bình | Average |
|
||||
| epic | 6.5–8 | Khá | Good |
|
||||
| legendary | 8–9 | Giỏi | Very good |
|
||||
| prismatic | 9–10 | Xuất sắc | Excellent |
|
||||
|
||||
Color + icon + text label — never color-only.
|
||||
|
||||
## Frontend key behaviors
|
||||
|
||||
- Query state lives in `App.jsx` and is synced to URL `?q=...` via `history.replaceState` — bookmarkable, shareable.
|
||||
- `SearchForm` is a controlled component that 300ms-debounces input changes and auto-triggers search once the query is long enough (3+ digits for SBD, 2+ chars for name).
|
||||
- Detection: all-digits → exact `so_bao_danh` match; ASCII-only → folded `ho_ten_ascii LIKE`; has Vietnamese diacritics → search both `ho_ten` and folded column.
|
||||
- Exactly 1 result → `StudentDetail` card; 0 or >1 → `ScoreTable`.
|
||||
- Global `/` key focuses the search input when not already typing.
|
||||
- Share button on the detail card uses Web Share API when present, else clipboard with a pre-formatted summary + deep-link URL.
|
||||
Reference in new issue
Block a user