mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-10-14 16:18:10 +00:00
tiennm99
c988bfafcf
The site credited 2016 to Bộ GD&ĐT, which those files were never fetched from. Both datasets come from published articles: 2016 from an aggregator on dtnt.bacninh.edu.vn listing one spreadsheet per exam cluster, 2017 from baotintuc.vn. The README, the architecture table and the web footer all repeated the ministry claim. The footer now shows each dataset's full article URL as a link rather than a bare host, so the citation can be checked. That needs overflow-wrap on the footer: the 2016 URL is 110 characters with no break opportunity and would otherwise scroll the page sideways on a phone. Also records that a full 2016 crawl has been run successfully. The host was marked unconfirmed and data/2016/ described as the only recoverable copy; both datasets are now rebuildable from source.
40 lines
1.5 KiB
Go
40 lines
1.5 KiB
Go
package sources
|
|
|
|
import (
|
|
"fmt"
|
|
|
|
"github.com/tiennm99/thptqg/crawler/internal/article"
|
|
)
|
|
|
|
// source2016 fetches the 2016 dataset: one spreadsheet per exam cluster
|
|
// (cụm thi), 4 .xls and 115 .xlsx.
|
|
//
|
|
// The article is a school site that aggregated every cluster's file. It is
|
|
// still online, and its links are read at crawl time like any other source.
|
|
//
|
|
// Dest keeps the server's own filename verbatim — a 32-hex content hash, the
|
|
// cluster slug, then a millisecond timestamp. Two reasons not to prettify it:
|
|
//
|
|
// - parser sorts inputs bytewise and inserts last-wins, so filenames decide
|
|
// which row survives a duplicate exam number. That is live here, not
|
|
// hypothetical: 877,464 source rows collapse to 877,461, so three rows'
|
|
// contents depend on this ordering.
|
|
// - parser/testdata/reader-fidelity-hashes.tsv is keyed by full path, and
|
|
// it is frozen — it was produced by the Rust reader, which no longer exists.
|
|
//
|
|
// Unlike 2017 this needs no transliteration: the name comes from the URL, so it
|
|
// is exact by construction rather than derived from a label.
|
|
var source2016 = Source{
|
|
ID: "2016",
|
|
Summary: "119 exam-cluster files (4 .xls + 115 .xlsx)",
|
|
Article: "https://dtnt.bacninh.edu.vn/tin-tuc/tin-tuc-su-kien/cong-bo-diem-thi-thptqg-2016-toan-bo-120-cum-thi-da-co-diem.html",
|
|
Exts: []string{".xls", ".xlsx"},
|
|
WantFiles: 119,
|
|
Dest: func(l article.Link) (string, error) {
|
|
if l.File == "" {
|
|
return "", fmt.Errorf("link %s has no filename", l.URL)
|
|
}
|
|
return l.File, nil
|
|
},
|
|
}
|