feat(dict): derive the corpus from undertheseanlp's wiktionary rows

Replace the 179 MB minhqnd SQLite aggregate with the 4.8 MB
undertheseanlp/dictionary JSONL, pinned by commit and SHA-256, reading
only rows tagged "wiktionary". The two other wordlists in that file are
never read: hongocduc is GPL and would force a relicense, tudientv is an
unlicensed derivative of a commercial dictionary.

build-dictionary gains --merged and --sources (names validated, default
wiktionary), a shared finish() tail, and meta rows for the source commit
and the sources kept and excluded. The SQLite --in path, its schema
auto-detection and their tests are removed. Fixture builds now record
that they carry no upstream data instead of inheriting a licence string.
The --min-words floor moves from 40,000 to 20,000; the corpus is 26,845
words, down from 48,216, all of the loss being words absent from the
2018 Wiktionary scrape. Capitalization is not a filter.

The data licence follows the source text: CC BY-SA 3.0 Unported, which
is what vi.wiktionary.org carried in 2018. LICENSE, NOTICE, ATTRIBUTION,
the README, the image docs, the builder's meta string and the in-game
footer all name Wiktionary tiếng Việt's contributors as the authors and
undertheseanlp as the intermediary. The Makefile/Dockerfile pin test now
also checks the commit the builder stamps into the database.
This commit is contained in:
tiennm99 committed 2026-09-08 16:37:34 +07:00
1 parent 341d4fd2e0
commit 39ef45730f
25 files changed
+927 -910

No files matched your search

+4 -4
View File
@@ -51,9 +51,9 @@ docker build -t noitu:latest .
docker run -p 8080:8080 noitu:latest
```
The build downloads the 179 MB upstream dictionary in a builder stage and
derives the ~3 MB database the game uses. Only the derived file is copied into
the final image, so the upstream release never ships. The result is a
The build downloads the 4.8 MB upstream wordlist in a builder stage and
derives the ~1.7 MB database the game uses. Only the derived file is copied into
the final image, so the upstream wordlist never ships. The result is a
distroless image of about 25 MB running as a non-root user.
Passing `--build-arg FIXTURE_DICT=1` builds the same image against the
@@ -62,7 +62,7 @@ exists so the image can be tested without the download; do not ship it.
### What travels with the data
The derived wordlist is CC BY-SA 4.0 while the code is Apache-2.0, so the image
The derived wordlist is CC BY-SA 3.0 while the code is Apache-2.0, so the image
carries `data/LICENSE`, `data/ATTRIBUTION.md` and `NOTICE` alongside it. CI
asserts all three are present, and that the upstream file is not. Removing them
would put the image out of compliance.