mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-10-20 00:15:51 +00:00
fix: correct the 2016 source attribution and link full article URLs
The site credited 2016 to Bộ GD&ĐT, which those files were never fetched from. Both datasets come from published articles: 2016 from an aggregator on dtnt.bacninh.edu.vn listing one spreadsheet per exam cluster, 2017 from baotintuc.vn. The README, the architecture table and the web footer all repeated the ministry claim. The footer now shows each dataset's full article URL as a link rather than a bare host, so the citation can be checked. That needs overflow-wrap on the footer: the 2016 URL is 110 characters with no break opportunity and would otherwise scroll the page sideways on a phone. Also records that a full 2016 crawl has been run successfully. The host was marked unconfirmed and data/2016/ described as the only recoverable copy; both datasets are now rebuildable from source.
This commit is contained in:
1 parent
3995e9d864
commit
c988bfafcf
9 files changed
+52
-31
No files matched your search
+13
-10
@@ -11,8 +11,10 @@ regexes are canonical and live in `parser/internal/schema/schema.go`.
|
||||
|
||||
| id | Files | Origin | Host live? |
|
||||
| --- | --- | --- | --- |
|
||||
| `2016` | 4 `.xls` + 115 `.xlsx` | aggregator article, 119 exam clusters | unconfirmed |
|
||||
| `2017` | 63 `.xls` | baotintuc.vn CDN | **yes** |
|
||||
| `2016` | 4 `.xls` + 115 `.xlsx` | `dtnt.bacninh.edu.vn` aggregator article, 119 exam clusters | **yes** |
|
||||
| `2017` | 63 `.xls` | `baotintuc.vn` article, files on its CDN | **yes** |
|
||||
|
||||
Full article URLs are below.
|
||||
|
||||
Crawling lives in `crawler/`, a separate Go module. It is never part of the
|
||||
build — the source files are committed, so a crawl only refreshes them. Both
|
||||
@@ -29,15 +31,16 @@ go -C crawler run ./cmd/crawl 2017 --list # list only, download nothing
|
||||
and its CDN is still serving the files.
|
||||
|
||||
**2016** comes from the aggregator article
|
||||
`cong-bo-diem-thi-thptqg-2016-toan-bo-120-cum-thi-da-co-diem.html`, served from
|
||||
a mirror — the site that first published it (`dtntbacgiang.edu.vn`) no longer
|
||||
resolves.
|
||||
`https://dtnt.bacninh.edu.vn/tin-tuc/tin-tuc-su-kien/cong-bo-diem-thi-thptqg-2016-toan-bo-120-cum-thi-da-co-diem.html`,
|
||||
which lists one spreadsheet per exam cluster. A full crawl against it has been
|
||||
run successfully and reproduces `data/2016/`, so the dataset is recoverable from
|
||||
source like 2017 is.
|
||||
|
||||
That mirror is **not reachable from every network.** It resolves to a Vietnamese
|
||||
address that times out from at least some hosts abroad, in which case the crawl
|
||||
stops with a connection error before downloading anything. `data/2016/` is
|
||||
therefore still the only confirmed copy: do not delete it on the assumption that
|
||||
a crawl can restore it.
|
||||
That host is **not reachable from every network**, though: it resolves to a
|
||||
Vietnamese address that times out from at least some hosts abroad, in which case
|
||||
the crawl stops with a connection error before downloading anything. That is a
|
||||
connectivity problem, not a missing dataset — retry from a network that can
|
||||
reach the host.
|
||||
|
||||
## How a source is defined
|
||||
|
||||
|
||||
@@ -35,18 +35,17 @@ running entirely in the browser and hosted for free on GitHub Pages. Covers the
|
||||
| id | Exam | Candidates | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| `2016` | 2016 | 877,461 | 119 files, three column layouts |
|
||||
| `2017` | 2017 | 861,068 | current generation, reproducible from source |
|
||||
| `2017` | 2017 | 861,068 | current generation of three publications |
|
||||
|
||||
Two further 2017 datasets (`2017-old`, `2017-old2`) were kept alongside these
|
||||
because the three publications disagreed. They have been removed; git history
|
||||
still has them.
|
||||
|
||||
Both datasets now have a crawler source. 2017 comes from the baotintuc.vn CDN,
|
||||
which is still live. 2016 comes from an aggregator article whose original host
|
||||
(`dtntbacgiang.edu.vn`) no longer resolves — its link list was recovered from
|
||||
the Internet Archive and is pointed at a mirror that is still online. See
|
||||
[data-pipeline](./data-pipeline.md#sources) for what that does and does not
|
||||
guarantee.
|
||||
which is still live. 2016 comes from an aggregator article on
|
||||
`dtnt.bacninh.edu.vn`, also still online. Both datasets have been crawled
|
||||
successfully, so either can be rebuilt from source — see
|
||||
[data-pipeline](./data-pipeline.md#sources) for the full article URLs.
|
||||
|
||||
## History
|
||||
|
||||
|
||||
@@ -51,8 +51,11 @@ that does not exist.
|
||||
|
||||
| id | Exam | Rows | Source |
|
||||
| --- | --- | --- | --- |
|
||||
| `2016` | 2016 | 877,461 | Bộ GD&ĐT |
|
||||
| `2017` | 2017 | 861,068 | baotintuc.vn |
|
||||
| `2016` | 2016 | 877,461 | `dtnt.bacninh.edu.vn` |
|
||||
| `2017` | 2017 | 861,068 | `baotintuc.vn` |
|
||||
|
||||
Full source URLs are in [data-pipeline](./data-pipeline.md#sources); the web
|
||||
footer links to them per dataset.
|
||||
|
||||
## Canonical schema
|
||||
|
||||
@@ -182,6 +185,6 @@ total descending.
|
||||
run out.
|
||||
- **`sql.js.org` dependency.** If that CDN is unreachable, the WASM fails to
|
||||
load. Self-hosting `sql-wasm.wasm` and updating `SQL_WASM_URL` in
|
||||
`use-sqlite.js` is the fix.
|
||||
`web/src/hooks/use-sqlite.js` is the fix.
|
||||
- **Excel format drift.** A new source file with an unseen header layout needs a
|
||||
new branch in `parser/internal/ingest/detect2016.go` or a new config.
|
||||
Reference in new issue
Block a user