tiennm99/noitu
1
0 Fork 0
mirror of https://github.com/tiennm99/noitu.git synced 2026-10-14 02:16:25 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(plans): survey viwiktionary and undertheseanlp as dictionary sources

Browse Source 
Measured against the shipped corpus rather than estimated: viwiktionary is
already a third of what we ship, and undertheseanlp is bigger and denser but
carries an unlicensed Vietlex derivative in one of its three branches.
This commit is contained in:
tiennm99 committed 2026-09-08 15:36:48 +07:00
1 parent 2f896584f8
commit 1c32cfa31a
2 files changed
+319
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

plans/reports/research-260908-1434-viwiktionary-as-dictionary-source.md
+170
View File
@@ -0,0 +1,170 @@
---
title: "Research: building our own dictionary DB from a viwiktionary dump"
date: 2026-09-08T14:34+07:00
type: research
status: complete
scope: research-only — no code touched
follows: research-260907-1650-vietnamese-dictionary-sources.md
---
# Research: can we build our own dictionary DB from a viwiktionary dump?
## Executive summary
**Technically yes, cheaply — but as a *replacement* it is a downgrade, and the answer to
"instead of the current database" is no.**
viwiktionary is already inside what we ship. The upstream `data/dictionary.db` carries a
per-row provenance table, and it says our corpus is four sources merged: `TVTD` 30,028
rows, `Wiktionary` (the vi edition) 17,365, `Wiktionary EN` 19,697, `tudientv.com` 3,421.
Of the 48,216 words in `data/noitu.db`, **15,924 (33%) trace to the viwiktionary source
and 32,292 (67%) do not.** A viwiktionary-only build would therefore have to *beat* our
current 16k contribution from that source by 3× before it merely broke even — and the
words it would add are the same crowd-sourced tail that produced the false-positive
problem the previous report measured.
**What is genuinely on offer is a self-controlled extraction, not more words.** wiktextract
has a real `vi` edition extractor (12 modules, ~138 KB) and **kaikki.org publishes a
prebuilt `viwiktionary/` dataset**, so we could take viwiktionary as a *primary* source
with POS and category metadata attached — which is exactly the filtering axis the current
opaque aggregate denies us. Proper nouns (`Danh từ riêng`), Nôm characters and
single-syllable morphemes become droppable by *label* rather than by heuristic.
**Recommendation: keep `minhqnd` as the breadth layer; add a self-extracted viwiktionary
as a second, POS-tagged source feeding the confidence tier from the previous report.** Do
not swap. And note the elephant: **`TVTD` alone is 24,369 of our shipped words (51%) and
its provenance is undocumented** — that is a bigger risk than anything viwiktionary poses.
---
## Methodology
- 5 web calls (budget cap): 1 search, 4 page fetches.
- Local evidence, measured directly this session: `data/noitu.db` and the 179 MB upstream
`data/dictionary.db`, both via sqlite.
- Repo evidence: `server/cmd/build-dictionary/{main,filter}.go`, `Makefile`.
- Key terms: `viwiktionary statistics`, `Mục từ tiếng Việt`, `wiktextract editions`,
`kaikki viwiktionary`.
---
## Finding 1 — viwiktionary's real size, in the only unit that matters
| Metric | Value | Source |
|---|---|---|
| Content pages | 346,778 | `vi.wiktionary.org/wiki/Special:Statistics` |
| Entries, all languages | 330,021 across **1,622 languages** | same |
| **Vietnamese-language entries** | **43,037** (`Thể loại:Mục từ tiếng Việt`) | category page |
| — of which Nôm characters | 1,731 (+ 4,380 entries carrying Nôm) | same |
| Total edits / active users / bots | 2,559,588 / **73** / 32 | Special:Statistics |
87% of viwiktionary's pages are entries, but **only ~13% of its entries are Vietnamese**;
the rest is foreign vocabulary written in Vietnamese. The usable ceiling is the 43,037,
minus Nôm characters, minus single-syllable words — which the game rejects outright
(`vietnamese.HasEnoughSyllables`, ≥2 syllables). Our corpus is 48,216 multi-syllable
words. **The ceiling of a viwiktionary-only build is plausibly below what we ship today,
and cannot exceed 43,037 even counting every entry we would have to throw away.**
73 active editors and 6.43 edits per page is a thinly maintained wiki. Freshness is not
the win here.
## Finding 2 — viwiktionary is already 33% of our corpus (measured)
`data/dictionary.db` has `sources(id, name)` and `words.source_id`. Measured this session,
Vietnamese rows only:
```
source raw multisyll shipped in noitu.db
TVTD 30028 24474 24369 <- 51% of our corpus
Wiktionary (vi) 17365 16216 15924 <- 33%
Wiktionary EN 19697 5121 4864 <- 10%
tudientv.com 3421 3064 3059 <- 6%
48216
```
Two things follow.
1. **A swap is a 2/3 cut.** Rebuilding from viwiktionary alone replaces 48,216 words with
~16k of known provenance plus whatever self-extraction recovers beyond the aggregate's
17,365-row snapshot. Even the optimistic case does not reach parity.
2. **The upstream deduplicates across sources, so agreement is unrecoverable.** Each word
carries exactly one `source_id` — the "only in viwiktionary" count equals its entire
set (16,216). We cannot ask "how many sources agree on this word?" of the current
upstream at all. Self-extracting viwiktionary is the only way to get that second
independent vote, which is the *actual* argument for doing it.
## Finding 3 — the extraction path is cheap and already exists
- `wiktextract` ships a per-edition `vi` extractor: `analyze_template, descendant,
etymology, example, linkage, models, page, pos, section_titles, sound, tags,
translation` (~138 KB). Real, not a stub; less mature than `en`.
- **kaikki.org publishes `https://kaikki.org/viwiktionary/`** — prebuilt, so no dump
download and no Lua-expanding extraction run is strictly required.
- Our builder already accepts a non-SQLite input: `--words <file>` (one word per line,
`#` comments), used today for the e2e fixture. A JSONL → wordlist reduction feeds it
with **zero builder changes**; a `--jsonl` input that keeps POS is a small addition.
- Everything downstream is unchanged: `accept()` normalizes, demands ≥2 syllables, and
already rejects digits, punctuation, non-Vietnamese alphabets and impossible syllables.
Cost is real but small: a reduction script, a pin, and one new attribution entry.
## Finding 4 — licensing is fine; pinning discipline is the catch
Wiktionary text is **CC BY-SA 4.0 (dual GFDL)** — same family as our current CC BY-SA 4.0
posture, share-alike either way. No relicensing of `noitu.db`, no `NOTICE` restructure;
`data/ATTRIBUTION.md` gains an entry naming viwiktionary and the dump/extraction date.
This is materially cleaner than the FVDP (GPL) option from the previous report.
The catch is reproducibility. `Makefile` pins `DICT_URL` + `DICT_SHA256`, asserted by
`web/tests/dictionary-source.test.js`. Wikimedia dumps are dated and immutable
(`viwiktionary/YYYYMMDD/`) so they pin cleanly; **kaikki.org publishes rolling files with
no versioned releases**, so pinning it means pinning a SHA-256 of a file that can be
replaced under the same URL. Prefer the dated Wikimedia dump + our own extraction run if
the pin guarantee matters more than convenience.
## Options
| | Option | Verdict |
|---|---|---|
| **A** | Replace `minhqnd` with viwiktionary-only | **No.** ~2/3 corpus cut, precision unchanged. |
| **B** | Self-extract viwiktionary as a *second* POS-tagged source, feeding the tier column | **Yes.** Buys the independent second vote and label-based proper-noun/Nôm rejection. |
| **C** | Status quo | Acceptable but leaves 51% of the corpus (`TVTD`) unprovenanced. |
Option B is the previous report's tiering design with viwiktionary supplying a vote that
is *ours* rather than an aggregate's — and it is the one source whose license is
unambiguously compatible.
## Recommended next steps
1. **Measure the ceiling before building anything.** Pull `https://kaikki.org/viwiktionary/`,
count Vietnamese lemmas with ≥2 syllables after our `accept()` rules. That single number
decides B outright: if it lands near 16k, viwiktionary adds a vote but no breadth; if it
lands near 35k, it is also a breadth layer.
2. Overlap it against our 48,216 — how many shipped words does viwiktionary confirm, and
how many of our unconfirmed words are in *no* independent list.
3. Check whether the extraction exposes `Danh từ riêng` / Nôm / morpheme labels usably.
That is the whole precision argument; if the labels are sparse, B loses most of its value.
4. **Chase `TVTD`.** It is 51% of what we ship and nothing in the repo says what it is. If
it is a scrape of a commercial dictionary, that outranks every question in this report.
5. Only then: `--jsonl` input + `tier` column + a dated Wikimedia dump pin.
## Sources
- [vi.wiktionary.org — Special:Statistics](https://vi.wiktionary.org/wiki/Special:Statistics)
- [Thể loại:Mục từ tiếng Việt](https://vi.wiktionary.org/wiki/Th%E1%BB%83_lo%E1%BA%A1i:M%E1%BB%A5c_t%E1%BB%AB_ti%E1%BA%BFng_Vi%E1%BB%87t)
- [kaikki.org — editions index, incl. `viwiktionary/`](https://kaikki.org/)
- [tatuylonen/wiktextract](https://github.com/tatuylonen/wiktextract) — `src/wiktextract/extractor/vi`
- [Wikimedia dumps](https://dumps.wikimedia.org/viwiktionary/)
- Local: `data/dictionary.db` (`sources`, `words`), `data/noitu.db` (`meta`, `words`)
## Unresolved questions
- **How many viwiktionary Vietnamese entries survive our ≥2-syllable filter?** Unmeasured;
the deciding number. Step 1 above.
- Is the `Wiktionary` source in `dictionary.db` really the vi edition? Inferred from its
name standing beside `Wiktionary EN`; the upstream does not document it.
- What is `TVTD`? Undocumented, and the majority of our corpus.
- Does kaikki's vi extraction carry per-entry categories, or only POS? Determines whether
proper nouns can be dropped by label.
- viwiktionary dump size and extraction runtime — not fetched (budget); only relevant if
we reject the kaikki prebuilt for pinning reasons.
plans/reports/research-260908-1507-undertheseanlp-dictionary.md
+149
View File
@@ -0,0 +1,149 @@
---
title: "Research: switching the corpus to undertheseanlp/dictionary"
date: 2026-09-08T15:07+07:00
type: research
status: complete
scope: research-only — no code touched, no shipped DB modified
follows: research-260908-1434-viwiktionary-as-dictionary-source.md
---
# Research: switch to `undertheseanlp/dictionary`?
## Executive summary
**Do not ship the merged list, and do not switch wholesale — but take one thing from it
today.** All four branches were downloaded and run through our real
`build-dictionary --words` filter this session; every number below is measured, not
estimated.
**The blocker is `tudientv`.** Its own branch README says copyright is *"Chưa rõ"*
(unknown) and that its primary source is *"Từ điển Soha … do Vietlex biên soạn (Hoàng Phê
chủ biên)"* — i.e. it is an unlicensed derivative of the commercial Vietlex/Hoàng Phê
dictionary that the first report already ruled out. That answers that report's open
question with a hard no. The merged 79,226-word list contains it, so the merged list is
not shippable. It only contributes 1,300 words we don't otherwise have.
**The votes are not a quality signal.** The merged file is JSONL with a per-word `source`
array, which looked like the confidence-tier input the earlier report designed for.
Measured, it isn't: `hà nội` has 2 votes, `cẩm xá` 2, `a mú sung` 2, `kháng đón` 2, while
`cánh diều` has 1 and `mặt trời` 2. A vote≥3 gate (14,346 words) drops `mặt trời` and
`con người`. The three sources are not independent enough to vote. **That design dies here.**
**The real find is capitalization.** undertheseanlp preserves case; `minhqnd` does not —
**0 of its 70,511 Vietnamese rows carry any uppercase**, so case is destroyed upstream and
unrecoverable. 4,773 entries appear *only* ever capitalized (`Hà Nội`, `Trương Công Định`,
`A Mú Sung`, `Cẩm Xá`), and `Mặt Trời` is not among them because it also occurs lowercase.
That set is a proper-noun blocklist, and **4,145 of our shipped 48,216 words are on it.**
**Recommendation: use undertheseanlp as a filter, not as the corpus.** Apply the blocklist
to what we already ship — 48,216 → 44,071, removing 4,145 place and person names, with the
DB staying CC BY-SA 4.0 because no undertheseanlp string is shipped. Adopting its words as
well is a real option but costs a **relicense of `noitu.db` to GPLv3**.
---
## Licensing, per branch (read from each branch's own README)
| Branch | Words | Declared license | Verdict |
|---|---|---|---|
| `master` (merged) | 79,226 | GPL-3.0 for *"mã nguồn"* (code); **data unstated** | Not shippable — contains `tudientv` |
| `hongocduc` | 73,172 | *"Dữ liệu được phân phối theo GNU GPL"* — data **is** GPL | Usable → forces GPLv3 downstream |
| `tudientv` | 36,533 | **"Bản quyền: Chưa rõ."** Primary source: Soha/Vietlex, Hoàng Phê chủ biên | **Never ship.** Unlicensed commercial derivative |
| `wiktionary` | 32,484 | scrape of vi.wiktionary.org, **2018-12-10** | CC BY-SA, but an 8-year-old snapshot |
Repo last pushed **2018-12-10**. `hongocduc` is Hồ Ngọc Đức's 2003 wordlist. So the whole
thing is a 2018 snapshot of mostly-2003 data — which is exactly why it misses modern
vocabulary (below).
## Measured yields, through our own `accept()` filter
| Corpus | Accepted words | Note |
|---|---|---|
| **current `noitu.db` (minhqnd)** | **48,216** | today |
| undertheseanlp, everything | 67,523 | includes unlicensed `tudientv` |
| everything − proper nouns | 62,892 | still includes `tudientv` |
| `hongocduc` + `wiktionary` | 65,907 | GPL + CC BY-SA |
| **`hongocduc` + `wiktionary` − proper nouns** | **61,276** | the only clean "switch" candidate |
| `hongocduc` alone | 62,571 (58,776 − proper nouns) | GPL |
| `wiktionary` branch alone | 26,845 (22,636 − proper nouns) | worse than extracting the live dump |
| **current − blocklist** | **44,071** | filter-only, license unchanged |
| **union(current, hnd+wik) − blocklist** | **69,049** | breadth + precision, needs GPLv3 |
Source composition of the 69,600 space-separated merged entries, for the record:
`hongocduc` only 26,533 · all three 14,346 · `hongocduc`+`tudientv` 14,198 ·
`hongocduc`+`wiktionary` 9,144 · `wiktionary` only 3,568 · `tudientv` only 1,720 ·
`tudientv`+`wiktionary` 91.
## What a straight switch would cost
Switching to `hongocduc`+`wiktionary`−proper-nouns (61,276) against today's 48,216:
overlap 36,298, **gained 24,978, lost 11,918**. The losses break down:
```
4,145 dropped deliberately as always-capitalized proper nouns <- wanted
1,300 present only in tudientv (unlicensed) <- unrecoverable cleanly
6,473 absent from undertheseanlp entirely <- the real cost
```
That last bucket is a mix of proper nouns minhqnd had already lowercased (`hồng kông`,
`mĩ đức tây`, `phù nam`, `cầu đỏ`) and **genuine modern vocabulary a 2003 list cannot
have**: `tế bào gốc`, `hằng số vũ trụ`, `đồng phạm giản đơn`, `bánh mì hoa cúc`,
`đèn hoa đăng`. minhqnd carries live Wiktionary; undertheseanlp is frozen at 2018.
## Blocklist validation
Against the always-capitalized set, 11 known-bad probes were all dropped
(`trương công định`, `trần danh án`, `tân an thạnh`, `cẩm xá`, `vĩnh điện`, `cam lâm`,
`hà nội`, `sài gòn`, `a mú sung`, `a lưới`, `kháng đón`) and 10 ordinary words all kept
(`học sinh`, `mặt trời`, `con người`, `bánh mì`, `cánh diều`, `xe đạp`, `giáo viên`,
`hoa hồng`, `tình yêu`, `nước mắm`). No false positives in the probe set.
It is not complete — it cannot catch a proper noun that never appears capitalized in
undertheseanlp, and 6,473 of our words are unknown to it entirely. It removes 4,145 of the
roughly 10–20% junk the first report sampled, which is the cheapest precision we have found.
## Options
| | Option | Words | License effect | Verdict |
|---|---|---|---|---|
| **A** | **Blocklist only** — filter today's corpus by undertheseanlp capitalization | 44,071 | **none**, stays CC BY-SA 4.0 | **Do this now.** Small, reversible, no new dependency in the shipped data |
| **B** | Union: current + `hongocduc`+`wiktionary`, blocklist applied | 69,049 | **`noitu.db` → GPLv3** | Best corpus. A deliberate license decision, not a technical one |
| **C** | Replace with `hongocduc`+`wiktionary` − proper nouns | 61,276 | `noitu.db` → GPLv3 | Worse than B for no gain — throws away 6,473 modern words |
| **D** | Ship the merged 79,226 list | 67,523 | unlicensed Vietlex derivative | **No.** |
GPL direction matters: CC BY-SA 4.0 → GPLv3 is permitted one-way, so a corpus mixing both
must go out as GPLv3. Filter-only use (A) ships no GPL string, so nothing changes.
## Recommended next steps
1. **A, now.** Vendor the derived blocklist (4,773 lowercased keys, ~90 KB) as a data file
with its provenance recorded, add `--blocklist <file>` to `build-dictionary`, count the
drops in the build log the way reject reasons already are, and pin the source commit
the way `DICT_SHA256` is pinned. It is a membership test, so `data/ATTRIBUTION.md` gains
a "used as a filter" note rather than a license change.
2. Spot-check 100 of the 4,145 drops by hand before it ships. A blocklist that eats a real
word is worse than the proper noun it removed.
3. Decide B separately, as a licensing question: is `noitu.db` allowed to be GPLv3? If yes,
B is the best corpus available from open data. If no, A is the ceiling and the
viwiktionary extractor from the previous report is the only breadth route left, since
Wiktionary text is CC BY-SA.
4. Drop `tudientv` from consideration permanently, and record why so nobody re-opens it.
## Sources
- [undertheseanlp/dictionary](https://github.com/undertheseanlp/dictionary) — branches
[`master`](https://github.com/undertheseanlp/dictionary/tree/master),
[`hongocduc`](https://github.com/undertheseanlp/dictionary/tree/hongocduc),
[`tudientv`](https://github.com/undertheseanlp/dictionary/tree/tudientv),
[`wiktionary`](https://github.com/undertheseanlp/dictionary/tree/wiktionary)
- [tudientv.com](http://tudientv.com/) — the `tudientv` branch's upstream
- Local: `data/noitu.db`, `data/dictionary.db`, `server/cmd/build-dictionary`
## Unresolved questions
- Is a GPLv3 `noitu.db` acceptable? Product/licensing call; decides option B.
- The 4,145 blocklist drops are unaudited. Sample them before shipping.
- 6,473 of our words are in no undertheseanlp list. Unknown split between junk and modern
vocabulary; a second signal (live viwiktionary) would separate them.
- `hongocduc` is distributed as GPL by a mirror, and the first report could not reach Hồ
Ngọc Đức's canonical site (404). The license claim rests on the mirror's README.