feat(dict): derive the corpus from kaikki.org's Wiktionary tiếng Việt export

Replace the pinned 2018 undertheseanlp wordlist with kaikki.org's current
wiktextract export of the Vietnamese Wiktionary, read with --kaikki. The
same authors and website, eight years fresher: 34,813 words instead of
26,845, with every graph metric up and bot game length unchanged.

The export is fetched fresh for every build and is not pinned, by the
owner's decision: kaikki keeps no dated snapshots, so a checksum would
break weekly. The builder therefore hashes the file as it streams it and
records source_sha256, source_rows and source_fetched_at in meta; the
fetch downloads to a .part name and renames on success; a truncated or
non-JSON body fails the build, and the --min-words floor rises to 30,000.
DICT_SHA256 and verify-dict are gone; the Makefile, Dockerfile and the
builder's URL constant are held in agreement by a test.

Current Wiktionary text is CC BY-SA 4.0, so the data licence returns to
4.0: data/LICENSE is restored, and NOTICE, ATTRIBUTION, README, the image
docs and the in-game footer credit Wiktionary tiếng Việt's contributors
and wiktextract/kaikki.org. builder_version becomes 3 for the changed
meta contract.
This commit is contained in:
tiennm99 committed 2026-09-08 17:17:58 +07:00
1 parent 5702e6f6cc
commit 0a5c5b06fe
23 files changed
+1017 -882

No files matched your search

+3 -3
View File
@@ -123,7 +123,7 @@ jobs:
docker build --build-arg FIXTURE_DICT=1 -t noitu:ci .
fi
# CC BY-SA 3.0 applies to the derived wordlist wherever it is
# CC BY-SA 4.0 applies to the derived wordlist wherever it is
# distributed, and an image is distribution. This is the assertion that
# the obligation actually shipped.
- name: The licence travels with the data
@@ -137,8 +137,8 @@ jobs:
grep -qx "$required" files.txt || { echo "missing from the image: $required"; exit 1; }
done
# The upstream wordlist must never reach the final image.
if grep -q 'words\.jsonl$' files.txt; then
# The upstream export must never reach the final image.
if grep -q '\.jsonl$' files.txt; then
echo "the upstream wordlist leaked into the image"
exit 1
fi