feat(dict): build the corpus and word meanings from the Wikimedia viwiktionary dump

reader for both wikitext dialects; `meanings(word, ord, pos, gloss)` table; `meaning_count`/`words_with_meaning`/`source_pages` in meta, `source_rows` gone, builder_version 5; `--dump`/`--min-pages` replace `--kaikki`; attribution names the dump and the definition excerpts; 36,200 words, 96.9% with a meaning, every kaikki word kept.
This commit is contained in:
tiennm99 committed 2026-09-08 22:48:26 +07:00
1 parent 557de1af94
commit 80f216d56b
25 files changed
+2338 -757

No files matched your search

+8 -7
View File
@@ -140,12 +140,13 @@ persistence, by design, in this version.
## Updating the dictionary
The wordlist is a build artifact, not runtime state, and the upstream export is
fetched fresh rather than pinned: rebuilding the image picks up whatever
kaikki.org currently serves, and the database's `meta` table records the
SHA-256 of the file it was built from. To update the dictionary, rebuild and
redeploy. To change the source itself, update `DICT_URL` in the `Dockerfile`,
the `Makefile` and the builder's constant (a test asserts the three agree),
then record what changed in `data/ATTRIBUTION.md`.
The dictionary is a build artifact, not runtime state, and the upstream dump
is fetched fresh rather than pinned: rebuilding the image picks up whatever
`dumps.wikimedia.org` currently serves under `viwiktionary/latest/`, which is
regenerated monthly, and the database's `meta` table records the SHA-256 of
the file it was built from. To update the dictionary, rebuild and redeploy.
To change the source itself, update `DICT_URL` in the `Dockerfile`, the
`Makefile` and the builder's constant (a test asserts the three agree), then
record what changed in `data/ATTRIBUTION.md`.
Nothing migrates, because nothing persists.