feat(dict): derive the corpus from undertheseanlp's wiktionary rows

Replace the 179 MB minhqnd SQLite aggregate with the 4.8 MB
undertheseanlp/dictionary JSONL, pinned by commit and SHA-256, reading
only rows tagged "wiktionary". The two other wordlists in that file are
never read: hongocduc is GPL and would force a relicense, tudientv is an
unlicensed derivative of a commercial dictionary.

build-dictionary gains --merged and --sources (names validated, default
wiktionary), a shared finish() tail, and meta rows for the source commit
and the sources kept and excluded. The SQLite --in path, its schema
auto-detection and their tests are removed. Fixture builds now record
that they carry no upstream data instead of inheriting a licence string.
The --min-words floor moves from 40,000 to 20,000; the corpus is 26,845
words, down from 48,216, all of the loss being words absent from the
2018 Wiktionary scrape. Capitalization is not a filter.

The data licence follows the source text: CC BY-SA 3.0 Unported, which
is what vi.wiktionary.org carried in 2018. LICENSE, NOTICE, ATTRIBUTION,
the README, the image docs, the builder's meta string and the in-game
footer all name Wiktionary tiếng Việt's contributors as the authors and
undertheseanlp as the intermediary. The Makefile/Dockerfile pin test now
also checks the commit the builder stamps into the database.
This commit is contained in:
tiennm99 committed 2026-09-08 16:37:34 +07:00
1 parent 341d4fd2e0
commit 39ef45730f
25 files changed
+927 -910

No files matched your search

+6 -6
View File
@@ -1,7 +1,7 @@
# Tests and packaging.
#
# Nothing here downloads the 179 MB upstream dictionary except the release
# image build. Everything else plays against the small database derived from
# Nothing here downloads the upstream wordlist except the release image
# build. Everything else plays against the small database derived from
# the checked-in word sample, which goes through the same builder the real one
# does.
#
@@ -123,7 +123,7 @@ jobs:
docker build --build-arg FIXTURE_DICT=1 -t noitu:ci .
fi
# CC BY-SA 4.0 applies to the derived wordlist wherever it is
# CC BY-SA 3.0 applies to the derived wordlist wherever it is
# distributed, and an image is distribution. This is the assertion that
# the obligation actually shipped.
- name: The licence travels with the data
@@ -137,9 +137,9 @@ jobs:
grep -qx "$required" files.txt || { echo "missing from the image: $required"; exit 1; }
done
# The 179 MB source must never reach the final image.
if grep -q 'dictionary\.db$' files.txt; then
echo "the upstream dictionary leaked into the image"
# The upstream wordlist must never reach the final image.
if grep -q 'words\.jsonl$' files.txt; then
echo "the upstream wordlist leaked into the image"
exit 1
fi
+1 -1
View File
@@ -2,7 +2,7 @@
# broken compatibility, and the committed generated code matches it.
#
# The suites that decode the cross-language fixtures run in ci.yml, where the
# rest of the tests are. Nothing here downloads the 179 MB upstream dictionary.
# rest of the tests are. Nothing here downloads the upstream wordlist.
name: proto
on: