diff --git a/plans/reports/research-260908-1434-viwiktionary-as-dictionary-source.md b/plans/reports/research-260908-1434-viwiktionary-as-dictionary-source.md new file mode 100644 index 0000000..a4a0531 --- /dev/null +++ b/plans/reports/research-260908-1434-viwiktionary-as-dictionary-source.md @@ -0,0 +1,170 @@ +--- +title: "Research: building our own dictionary DB from a viwiktionary dump" +date: 2026-09-08T14:34+07:00 +type: research +status: complete +scope: research-only — no code touched +follows: research-260907-1650-vietnamese-dictionary-sources.md +--- + +# Research: can we build our own dictionary DB from a viwiktionary dump? + +## Executive summary + +**Technically yes, cheaply — but as a *replacement* it is a downgrade, and the answer to +"instead of the current database" is no.** + +viwiktionary is already inside what we ship. The upstream `data/dictionary.db` carries a +per-row provenance table, and it says our corpus is four sources merged: `TVTD` 30,028 +rows, `Wiktionary` (the vi edition) 17,365, `Wiktionary EN` 19,697, `tudientv.com` 3,421. +Of the 48,216 words in `data/noitu.db`, **15,924 (33%) trace to the viwiktionary source +and 32,292 (67%) do not.** A viwiktionary-only build would therefore have to *beat* our +current 16k contribution from that source by 3× before it merely broke even — and the +words it would add are the same crowd-sourced tail that produced the false-positive +problem the previous report measured. + +**What is genuinely on offer is a self-controlled extraction, not more words.** wiktextract +has a real `vi` edition extractor (12 modules, ~138 KB) and **kaikki.org publishes a +prebuilt `viwiktionary/` dataset**, so we could take viwiktionary as a *primary* source +with POS and category metadata attached — which is exactly the filtering axis the current +opaque aggregate denies us. Proper nouns (`Danh từ riêng`), Nôm characters and +single-syllable morphemes become droppable by *label* rather than by heuristic. + +**Recommendation: keep `minhqnd` as the breadth layer; add a self-extracted viwiktionary +as a second, POS-tagged source feeding the confidence tier from the previous report.** Do +not swap. And note the elephant: **`TVTD` alone is 24,369 of our shipped words (51%) and +its provenance is undocumented** — that is a bigger risk than anything viwiktionary poses. + +--- + +## Methodology + +- 5 web calls (budget cap): 1 search, 4 page fetches. +- Local evidence, measured directly this session: `data/noitu.db` and the 179 MB upstream + `data/dictionary.db`, both via sqlite. +- Repo evidence: `server/cmd/build-dictionary/{main,filter}.go`, `Makefile`. +- Key terms: `viwiktionary statistics`, `Mục từ tiếng Việt`, `wiktextract editions`, + `kaikki viwiktionary`. + +--- + +## Finding 1 — viwiktionary's real size, in the only unit that matters + +| Metric | Value | Source | +|---|---|---| +| Content pages | 346,778 | `vi.wiktionary.org/wiki/Special:Statistics` | +| Entries, all languages | 330,021 across **1,622 languages** | same | +| **Vietnamese-language entries** | **43,037** (`Thể loại:Mục từ tiếng Việt`) | category page | +| — of which Nôm characters | 1,731 (+ 4,380 entries carrying Nôm) | same | +| Total edits / active users / bots | 2,559,588 / **73** / 32 | Special:Statistics | + +87% of viwiktionary's pages are entries, but **only ~13% of its entries are Vietnamese**; +the rest is foreign vocabulary written in Vietnamese. The usable ceiling is the 43,037, +minus Nôm characters, minus single-syllable words — which the game rejects outright +(`vietnamese.HasEnoughSyllables`, ≥2 syllables). Our corpus is 48,216 multi-syllable +words. **The ceiling of a viwiktionary-only build is plausibly below what we ship today, +and cannot exceed 43,037 even counting every entry we would have to throw away.** + +73 active editors and 6.43 edits per page is a thinly maintained wiki. Freshness is not +the win here. + +## Finding 2 — viwiktionary is already 33% of our corpus (measured) + +`data/dictionary.db` has `sources(id, name)` and `words.source_id`. Measured this session, +Vietnamese rows only: + +``` +source raw multisyll shipped in noitu.db +TVTD 30028 24474 24369 <- 51% of our corpus +Wiktionary (vi) 17365 16216 15924 <- 33% +Wiktionary EN 19697 5121 4864 <- 10% +tudientv.com 3421 3064 3059 <- 6% + 48216 +``` + +Two things follow. + +1. **A swap is a 2/3 cut.** Rebuilding from viwiktionary alone replaces 48,216 words with + ~16k of known provenance plus whatever self-extraction recovers beyond the aggregate's + 17,365-row snapshot. Even the optimistic case does not reach parity. +2. **The upstream deduplicates across sources, so agreement is unrecoverable.** Each word + carries exactly one `source_id` — the "only in viwiktionary" count equals its entire + set (16,216). We cannot ask "how many sources agree on this word?" of the current + upstream at all. Self-extracting viwiktionary is the only way to get that second + independent vote, which is the *actual* argument for doing it. + +## Finding 3 — the extraction path is cheap and already exists + +- `wiktextract` ships a per-edition `vi` extractor: `analyze_template, descendant, + etymology, example, linkage, models, page, pos, section_titles, sound, tags, + translation` (~138 KB). Real, not a stub; less mature than `en`. +- **kaikki.org publishes `https://kaikki.org/viwiktionary/`** — prebuilt, so no dump + download and no Lua-expanding extraction run is strictly required. +- Our builder already accepts a non-SQLite input: `--words ` (one word per line, + `#` comments), used today for the e2e fixture. A JSONL → wordlist reduction feeds it + with **zero builder changes**; a `--jsonl` input that keeps POS is a small addition. +- Everything downstream is unchanged: `accept()` normalizes, demands ≥2 syllables, and + already rejects digits, punctuation, non-Vietnamese alphabets and impossible syllables. + +Cost is real but small: a reduction script, a pin, and one new attribution entry. + +## Finding 4 — licensing is fine; pinning discipline is the catch + +Wiktionary text is **CC BY-SA 4.0 (dual GFDL)** — same family as our current CC BY-SA 4.0 +posture, share-alike either way. No relicensing of `noitu.db`, no `NOTICE` restructure; +`data/ATTRIBUTION.md` gains an entry naming viwiktionary and the dump/extraction date. +This is materially cleaner than the FVDP (GPL) option from the previous report. + +The catch is reproducibility. `Makefile` pins `DICT_URL` + `DICT_SHA256`, asserted by +`web/tests/dictionary-source.test.js`. Wikimedia dumps are dated and immutable +(`viwiktionary/YYYYMMDD/`) so they pin cleanly; **kaikki.org publishes rolling files with +no versioned releases**, so pinning it means pinning a SHA-256 of a file that can be +replaced under the same URL. Prefer the dated Wikimedia dump + our own extraction run if +the pin guarantee matters more than convenience. + +## Options + +| | Option | Verdict | +|---|---|---| +| **A** | Replace `minhqnd` with viwiktionary-only | **No.** ~2/3 corpus cut, precision unchanged. | +| **B** | Self-extract viwiktionary as a *second* POS-tagged source, feeding the tier column | **Yes.** Buys the independent second vote and label-based proper-noun/Nôm rejection. | +| **C** | Status quo | Acceptable but leaves 51% of the corpus (`TVTD`) unprovenanced. | + +Option B is the previous report's tiering design with viwiktionary supplying a vote that +is *ours* rather than an aggregate's — and it is the one source whose license is +unambiguously compatible. + +## Recommended next steps + +1. **Measure the ceiling before building anything.** Pull `https://kaikki.org/viwiktionary/`, + count Vietnamese lemmas with ≥2 syllables after our `accept()` rules. That single number + decides B outright: if it lands near 16k, viwiktionary adds a vote but no breadth; if it + lands near 35k, it is also a breadth layer. +2. Overlap it against our 48,216 — how many shipped words does viwiktionary confirm, and + how many of our unconfirmed words are in *no* independent list. +3. Check whether the extraction exposes `Danh từ riêng` / Nôm / morpheme labels usably. + That is the whole precision argument; if the labels are sparse, B loses most of its value. +4. **Chase `TVTD`.** It is 51% of what we ship and nothing in the repo says what it is. If + it is a scrape of a commercial dictionary, that outranks every question in this report. +5. Only then: `--jsonl` input + `tier` column + a dated Wikimedia dump pin. + +## Sources + +- [vi.wiktionary.org — Special:Statistics](https://vi.wiktionary.org/wiki/Special:Statistics) +- [Thể loại:Mục từ tiếng Việt](https://vi.wiktionary.org/wiki/Th%E1%BB%83_lo%E1%BA%A1i:M%E1%BB%A5c_t%E1%BB%AB_ti%E1%BA%BFng_Vi%E1%BB%87t) +- [kaikki.org — editions index, incl. `viwiktionary/`](https://kaikki.org/) +- [tatuylonen/wiktextract](https://github.com/tatuylonen/wiktextract) — `src/wiktextract/extractor/vi` +- [Wikimedia dumps](https://dumps.wikimedia.org/viwiktionary/) +- Local: `data/dictionary.db` (`sources`, `words`), `data/noitu.db` (`meta`, `words`) + +## Unresolved questions + +- **How many viwiktionary Vietnamese entries survive our ≥2-syllable filter?** Unmeasured; + the deciding number. Step 1 above. +- Is the `Wiktionary` source in `dictionary.db` really the vi edition? Inferred from its + name standing beside `Wiktionary EN`; the upstream does not document it. +- What is `TVTD`? Undocumented, and the majority of our corpus. +- Does kaikki's vi extraction carry per-entry categories, or only POS? Determines whether + proper nouns can be dropped by label. +- viwiktionary dump size and extraction runtime — not fetched (budget); only relevant if + we reject the kaikki prebuilt for pinning reasons. diff --git a/plans/reports/research-260908-1507-undertheseanlp-dictionary.md b/plans/reports/research-260908-1507-undertheseanlp-dictionary.md new file mode 100644 index 0000000..8b88bd8 --- /dev/null +++ b/plans/reports/research-260908-1507-undertheseanlp-dictionary.md @@ -0,0 +1,149 @@ +--- +title: "Research: switching the corpus to undertheseanlp/dictionary" +date: 2026-09-08T15:07+07:00 +type: research +status: complete +scope: research-only — no code touched, no shipped DB modified +follows: research-260908-1434-viwiktionary-as-dictionary-source.md +--- + +# Research: switch to `undertheseanlp/dictionary`? + +## Executive summary + +**Do not ship the merged list, and do not switch wholesale — but take one thing from it +today.** All four branches were downloaded and run through our real +`build-dictionary --words` filter this session; every number below is measured, not +estimated. + +**The blocker is `tudientv`.** Its own branch README says copyright is *"Chưa rõ"* +(unknown) and that its primary source is *"Từ điển Soha … do Vietlex biên soạn (Hoàng Phê +chủ biên)"* — i.e. it is an unlicensed derivative of the commercial Vietlex/Hoàng Phê +dictionary that the first report already ruled out. That answers that report's open +question with a hard no. The merged 79,226-word list contains it, so the merged list is +not shippable. It only contributes 1,300 words we don't otherwise have. + +**The votes are not a quality signal.** The merged file is JSONL with a per-word `source` +array, which looked like the confidence-tier input the earlier report designed for. +Measured, it isn't: `hà nội` has 2 votes, `cẩm xá` 2, `a mú sung` 2, `kháng đón` 2, while +`cánh diều` has 1 and `mặt trời` 2. A vote≥3 gate (14,346 words) drops `mặt trời` and +`con người`. The three sources are not independent enough to vote. **That design dies here.** + +**The real find is capitalization.** undertheseanlp preserves case; `minhqnd` does not — +**0 of its 70,511 Vietnamese rows carry any uppercase**, so case is destroyed upstream and +unrecoverable. 4,773 entries appear *only* ever capitalized (`Hà Nội`, `Trương Công Định`, +`A Mú Sung`, `Cẩm Xá`), and `Mặt Trời` is not among them because it also occurs lowercase. +That set is a proper-noun blocklist, and **4,145 of our shipped 48,216 words are on it.** + +**Recommendation: use undertheseanlp as a filter, not as the corpus.** Apply the blocklist +to what we already ship — 48,216 → 44,071, removing 4,145 place and person names, with the +DB staying CC BY-SA 4.0 because no undertheseanlp string is shipped. Adopting its words as +well is a real option but costs a **relicense of `noitu.db` to GPLv3**. + +--- + +## Licensing, per branch (read from each branch's own README) + +| Branch | Words | Declared license | Verdict | +|---|---|---|---| +| `master` (merged) | 79,226 | GPL-3.0 for *"mã nguồn"* (code); **data unstated** | Not shippable — contains `tudientv` | +| `hongocduc` | 73,172 | *"Dữ liệu được phân phối theo GNU GPL"* — data **is** GPL | Usable → forces GPLv3 downstream | +| `tudientv` | 36,533 | **"Bản quyền: Chưa rõ."** Primary source: Soha/Vietlex, Hoàng Phê chủ biên | **Never ship.** Unlicensed commercial derivative | +| `wiktionary` | 32,484 | scrape of vi.wiktionary.org, **2018-12-10** | CC BY-SA, but an 8-year-old snapshot | + +Repo last pushed **2018-12-10**. `hongocduc` is Hồ Ngọc Đức's 2003 wordlist. So the whole +thing is a 2018 snapshot of mostly-2003 data — which is exactly why it misses modern +vocabulary (below). + +## Measured yields, through our own `accept()` filter + +| Corpus | Accepted words | Note | +|---|---|---| +| **current `noitu.db` (minhqnd)** | **48,216** | today | +| undertheseanlp, everything | 67,523 | includes unlicensed `tudientv` | +| everything − proper nouns | 62,892 | still includes `tudientv` | +| `hongocduc` + `wiktionary` | 65,907 | GPL + CC BY-SA | +| **`hongocduc` + `wiktionary` − proper nouns** | **61,276** | the only clean "switch" candidate | +| `hongocduc` alone | 62,571 (58,776 − proper nouns) | GPL | +| `wiktionary` branch alone | 26,845 (22,636 − proper nouns) | worse than extracting the live dump | +| **current − blocklist** | **44,071** | filter-only, license unchanged | +| **union(current, hnd+wik) − blocklist** | **69,049** | breadth + precision, needs GPLv3 | + +Source composition of the 69,600 space-separated merged entries, for the record: +`hongocduc` only 26,533 · all three 14,346 · `hongocduc`+`tudientv` 14,198 · +`hongocduc`+`wiktionary` 9,144 · `wiktionary` only 3,568 · `tudientv` only 1,720 · +`tudientv`+`wiktionary` 91. + +## What a straight switch would cost + +Switching to `hongocduc`+`wiktionary`−proper-nouns (61,276) against today's 48,216: +overlap 36,298, **gained 24,978, lost 11,918**. The losses break down: + +``` +4,145 dropped deliberately as always-capitalized proper nouns <- wanted +1,300 present only in tudientv (unlicensed) <- unrecoverable cleanly +6,473 absent from undertheseanlp entirely <- the real cost +``` + +That last bucket is a mix of proper nouns minhqnd had already lowercased (`hồng kông`, +`mĩ đức tây`, `phù nam`, `cầu đỏ`) and **genuine modern vocabulary a 2003 list cannot +have**: `tế bào gốc`, `hằng số vũ trụ`, `đồng phạm giản đơn`, `bánh mì hoa cúc`, +`đèn hoa đăng`. minhqnd carries live Wiktionary; undertheseanlp is frozen at 2018. + +## Blocklist validation + +Against the always-capitalized set, 11 known-bad probes were all dropped +(`trương công định`, `trần danh án`, `tân an thạnh`, `cẩm xá`, `vĩnh điện`, `cam lâm`, +`hà nội`, `sài gòn`, `a mú sung`, `a lưới`, `kháng đón`) and 10 ordinary words all kept +(`học sinh`, `mặt trời`, `con người`, `bánh mì`, `cánh diều`, `xe đạp`, `giáo viên`, +`hoa hồng`, `tình yêu`, `nước mắm`). No false positives in the probe set. + +It is not complete — it cannot catch a proper noun that never appears capitalized in +undertheseanlp, and 6,473 of our words are unknown to it entirely. It removes 4,145 of the +roughly 10–20% junk the first report sampled, which is the cheapest precision we have found. + +## Options + +| | Option | Words | License effect | Verdict | +|---|---|---|---|---| +| **A** | **Blocklist only** — filter today's corpus by undertheseanlp capitalization | 44,071 | **none**, stays CC BY-SA 4.0 | **Do this now.** Small, reversible, no new dependency in the shipped data | +| **B** | Union: current + `hongocduc`+`wiktionary`, blocklist applied | 69,049 | **`noitu.db` → GPLv3** | Best corpus. A deliberate license decision, not a technical one | +| **C** | Replace with `hongocduc`+`wiktionary` − proper nouns | 61,276 | `noitu.db` → GPLv3 | Worse than B for no gain — throws away 6,473 modern words | +| **D** | Ship the merged 79,226 list | 67,523 | unlicensed Vietlex derivative | **No.** | + +GPL direction matters: CC BY-SA 4.0 → GPLv3 is permitted one-way, so a corpus mixing both +must go out as GPLv3. Filter-only use (A) ships no GPL string, so nothing changes. + +## Recommended next steps + +1. **A, now.** Vendor the derived blocklist (4,773 lowercased keys, ~90 KB) as a data file + with its provenance recorded, add `--blocklist ` to `build-dictionary`, count the + drops in the build log the way reject reasons already are, and pin the source commit + the way `DICT_SHA256` is pinned. It is a membership test, so `data/ATTRIBUTION.md` gains + a "used as a filter" note rather than a license change. +2. Spot-check 100 of the 4,145 drops by hand before it ships. A blocklist that eats a real + word is worse than the proper noun it removed. +3. Decide B separately, as a licensing question: is `noitu.db` allowed to be GPLv3? If yes, + B is the best corpus available from open data. If no, A is the ceiling and the + viwiktionary extractor from the previous report is the only breadth route left, since + Wiktionary text is CC BY-SA. +4. Drop `tudientv` from consideration permanently, and record why so nobody re-opens it. + +## Sources + +- [undertheseanlp/dictionary](https://github.com/undertheseanlp/dictionary) — branches + [`master`](https://github.com/undertheseanlp/dictionary/tree/master), + [`hongocduc`](https://github.com/undertheseanlp/dictionary/tree/hongocduc), + [`tudientv`](https://github.com/undertheseanlp/dictionary/tree/tudientv), + [`wiktionary`](https://github.com/undertheseanlp/dictionary/tree/wiktionary) +- [tudientv.com](http://tudientv.com/) — the `tudientv` branch's upstream +- Local: `data/noitu.db`, `data/dictionary.db`, `server/cmd/build-dictionary` + +## Unresolved questions + +- Is a GPLv3 `noitu.db` acceptable? Product/licensing call; decides option B. +- The 4,145 blocklist drops are unaudited. Sample them before shipping. +- 6,473 of our words are in no undertheseanlp list. Unknown split between junk and modern + vocabulary; a second signal (live viwiktionary) would separate them. +- `hongocduc` is distributed as GPL by a mirror, and the first report could not reach Hồ + Ngọc Đức's canonical site (404). The license claim rests on the mirror's README.