mirror of
https://github.com/tiennm99/noitu.git
synced 2026-10-15 04:18:50 +00:00
feat(dict): build the corpus and word meanings from the Wikimedia viwiktionary dump
reader for both wikitext dialects; `meanings(word, ord, pos, gloss)` table; `meaning_count`/`words_with_meaning`/`source_pages` in meta, `source_rows` gone, builder_version 5; `--dump`/`--min-pages` replace `--kaikki`; attribution names the dump and the definition excerpts; 36,200 words, 96.9% with a meaning, every kaikki word kept.
This commit is contained in:
1 parent
557de1af94
commit
80f216d56b
25 files changed
+2338
-757
No files matched your search
+8
-7
@@ -140,12 +140,13 @@ persistence, by design, in this version.
|
||||
|
||||
## Updating the dictionary
|
||||
|
||||
The wordlist is a build artifact, not runtime state, and the upstream export is
|
||||
fetched fresh rather than pinned: rebuilding the image picks up whatever
|
||||
kaikki.org currently serves, and the database's `meta` table records the
|
||||
SHA-256 of the file it was built from. To update the dictionary, rebuild and
|
||||
redeploy. To change the source itself, update `DICT_URL` in the `Dockerfile`,
|
||||
the `Makefile` and the builder's constant (a test asserts the three agree),
|
||||
then record what changed in `data/ATTRIBUTION.md`.
|
||||
The dictionary is a build artifact, not runtime state, and the upstream dump
|
||||
is fetched fresh rather than pinned: rebuilding the image picks up whatever
|
||||
`dumps.wikimedia.org` currently serves under `viwiktionary/latest/`, which is
|
||||
regenerated monthly, and the database's `meta` table records the SHA-256 of
|
||||
the file it was built from. To update the dictionary, rebuild and redeploy.
|
||||
To change the source itself, update `DICT_URL` in the `Dockerfile`, the
|
||||
`Makefile` and the builder's constant (a test asserts the three agree), then
|
||||
record what changed in `data/ATTRIBUTION.md`.
|
||||
|
||||
Nothing migrates, because nothing persists.
|
||||
Reference in new issue
Block a user