mirror of
https://github.com/tiennm99/noitu.git
synced 2026-10-19 00:15:08 +00:00
feat(dict): derive the corpus from undertheseanlp's wiktionary rows
Replace the 179 MB minhqnd SQLite aggregate with the 4.8 MB undertheseanlp/dictionary JSONL, pinned by commit and SHA-256, reading only rows tagged "wiktionary". The two other wordlists in that file are never read: hongocduc is GPL and would force a relicense, tudientv is an unlicensed derivative of a commercial dictionary. build-dictionary gains --merged and --sources (names validated, default wiktionary), a shared finish() tail, and meta rows for the source commit and the sources kept and excluded. The SQLite --in path, its schema auto-detection and their tests are removed. Fixture builds now record that they carry no upstream data instead of inheriting a licence string. The --min-words floor moves from 40,000 to 20,000; the corpus is 26,845 words, down from 48,216, all of the loss being words absent from the 2018 Wiktionary scrape. Capitalization is not a filter. The data licence follows the source text: CC BY-SA 3.0 Unported, which is what vi.wiktionary.org carried in 2018. LICENSE, NOTICE, ATTRIBUTION, the README, the image docs, the builder's meta string and the in-game footer all name Wiktionary tiếng Việt's contributors as the authors and undertheseanlp as the intermediary. The Makefile/Dockerfile pin test now also checks the commit the builder stamps into the database.
This commit is contained in:
1 parent
341d4fd2e0
commit
39ef45730f
25 files changed
+927
-910
No files matched your search
@@ -2,7 +2,7 @@
|
||||
# broken compatibility, and the committed generated code matches it.
|
||||
#
|
||||
# The suites that decode the cross-language fixtures run in ci.yml, where the
|
||||
# rest of the tests are. Nothing here downloads the 179 MB upstream dictionary.
|
||||
# rest of the tests are. Nothing here downloads the upstream wordlist.
|
||||
name: proto
|
||||
|
||||
on:
|
||||
|
||||
Reference in new issue
Block a user