diff --git a/plans/260908-1525-dictionary-corpus-switch/phase-01-start.md b/plans/260908-1525-dictionary-corpus-switch/phase-01-start.md new file mode 100644 index 0000000..e1a9107 --- /dev/null +++ b/plans/260908-1525-dictionary-corpus-switch/phase-01-start.md @@ -0,0 +1,120 @@ +--- +phase: 1 +title: "Phase 1: Read the merged list" +status: todo +priority: P1 +effort: "3h" +dependencies: [] +--- + +# Phase 1: Read the merged list + +## Overview + +Teach `build-dictionary` a third input: undertheseanlp's merged JSONL. It selects words by +source membership, drops entries that appear only ever capitalized, and hands the survivors +to the normalization and filtering path that already exists. + +## Requirements + +- Functional: read `{"text": "...", "source": ["hongocduc", ...]}` lines; keep a word when + its sources intersect the allowed set; drop a word when every raw form of it, across the + allowed sources, begins with an uppercase letter. +- Functional: `tudientv` rows are skipped before any decision is made about a word — + neither its words nor its capitalization inform the output. +- Functional: the proper-noun drop is counted and printed beside the existing reject + reasons, so the build log says what the source contained. +- Non-functional: streaming line-by-line, one pass to gather, one pass to decide. The file + is 4.8 MB; nothing here needs to be clever. +- Non-functional: everything downstream — `accept()`, syllable indexing, alias generation, + `verify()`, `meta` — is reused unchanged. + +## Architecture + +Two maps keyed by the lowercased, whitespace-collapsed word: + +``` +forms[key] -> set of raw spellings seen in allowed sources ("Mặt Trời", "mặt trời") +sources[key] -> set of allowed sources containing it {"hongocduc", "wiktionary"} +``` + +A key survives when `sources[key]` is non-empty and at least one entry in `forms[key]` +does not start with an uppercase letter. `Mặt Trời` survives on the strength of its +lowercase twin; `Hà Nội`, `Trương Công Định` and `A Mú Sung` have no lowercase form and go. + +Surviving keys are then fed to the same `accept()` the SQLite path feeds, so the ≥2 +syllable rule, the digit/punctuation rejections and the Vietnamese phonotactic check all +apply as they do today. + +Case evidence deliberately comes from allowed sources only. The alternative — reading +`tudientv` rows for their capitalization while refusing their words — would be more +accurate and would undercut the claim that we do not use that data. Phase 3 measures what +the stricter choice costs. + +## Related Code Files + +- Create: `server/cmd/build-dictionary/merged_list.go` +- Create: `server/cmd/build-dictionary/merged_list_test.go` +- Modify: `server/cmd/build-dictionary/main.go` — `--merged` and `--sources` flags, dispatch + in `run()`, `sourceURL`/`sourceLicense` constants, `meta` rows, package doc comment +- Modify: `server/cmd/build-dictionary/filter.go` — add the `rejectProperNoun` reason so + the drop is reported through the existing counter, not a bespoke log line + +## Implementation Steps + +1. Add `rejectProperNoun rejectReason = "only ever capitalized"` to `filter.go`. Leave + `accept()` alone — the case decision happens before it, on evidence `accept()` cannot + see, since it lowercases. +2. Write `merged_list.go`: decode with `encoding/json` per line (`bufio.Scanner` with a + raised buffer — the longest line is short, but a scanner that silently truncates is a + bad way to lose words), build the two maps, apply the rule, return words plus a count + per reject reason. +3. Add `--merged ` and `--sources hongocduc,wiktionary` to `main.go`. Unknown source + names are an error, not a silent no-op: a typo in `--sources` must not quietly ship an + empty or wrong corpus. +4. Dispatch in `run()`: `--merged` takes precedence in the same shape `--words` already + does. Refuse `--merged` together with `--in`/`--words` rather than picking one. +5. Point `sourceURL`, `sourceLicense` and the package doc comment at the new source. Write + `source_commit`, `sources_kept`, `sources_excluded` and `proper_nouns_dropped` into + `meta` — provenance in the artifact, not only in a markdown file. +6. Tests in `merged_list_test.go`, table-driven, over hand-written JSONL fixtures: + - `Mặt Trời` + `mặt trời` → kept + - `Hà Nội` alone → dropped as a proper noun + - a `tudientv`-only word → absent from the output + - a word whose only lowercase form is in `tudientv` → dropped (the documented cost) + - `học sinh` in all three → kept once, not thrice + - a malformed line → a clear error naming the line number, not a skipped word +7. Run against the real file and record the counts. Compare to the report's 61,276, which + used all-source case evidence; explain the delta rather than adjusting the number. + +## Success Criteria + +- [ ] `go test ./cmd/build-dictionary/` passes, new tests included. +- [ ] Building from the real merged file accepts >55,000 words and logs a proper-noun drop + count in the low thousands. +- [ ] Every one of the 11 known-bad probes from the report is absent from the output + (`trương công định`, `trần danh án`, `tân an thạnh`, `cẩm xá`, `vĩnh điện`, + `cam lâm`, `hà nội`, `sài gòn`, `a mú sung`, `a lưới`, `kháng đón`). +- [ ] All 10 ordinary-word probes are present (`học sinh`, `mặt trời`, `con người`, + `bánh mì`, `cánh diều`, `xe đạp`, `giáo viên`, `hoa hồng`, `tình yêu`, `nước mắm`). +- [ ] `--sources hongocduc` alone and `--sources hongocduc,wiktionary` both build, with the + first strictly smaller. +- [ ] No word in the output is reachable only through `tudientv`. + +## Risk Assessment + +**The capitalization rule eats real words.** The probe set is 21 words; the rule fires on +thousands. Signal: Phase 3's hand audit finds common words among the drops. Response: +narrow the rule to require every syllable capitalized (`Hà Nội`, not `Kháng đón`), re-audit, +and if that still misfires, stop dropping and keep the count as a log line only — a corpus +with proper nouns in it is what we ship today and is survivable. + +**Allowed-sources-only case evidence drops more than expected.** Signal: the drop count +comes in far above the report's 4,145. Response: measure how many drops have a lowercase +form only in `tudientv`; if that is a large share, reconsider reading `tudientv` for case +evidence alone and say so plainly in `ATTRIBUTION.md` rather than hiding it. + +**A JSON schema surprise.** The file is 8 years old and unversioned; a row could carry an +unexpected shape. Signal: decode errors on the real file. Response: the error names the +line — inspect it, and only then decide between a tolerant skip with a counted reason and +a hard failure. Silent skipping is not an option. diff --git a/plans/260908-1525-dictionary-corpus-switch/phase-02-pin-the-source.md b/plans/260908-1525-dictionary-corpus-switch/phase-02-pin-the-source.md new file mode 100644 index 0000000..19a1b09 --- /dev/null +++ b/plans/260908-1525-dictionary-corpus-switch/phase-02-pin-the-source.md @@ -0,0 +1,85 @@ +--- +phase: 2 +title: "Phase 2: Pin the source" +status: todo +priority: P1 +effort: "2h" +dependencies: [1] +--- + +# Phase 2: Pin the source + +## Overview + +Point the build machinery at the new file: pinned by commit and checksum, fetched by +`make fetch-dict` and by the Docker dictionary stage, with the existing agreement test +still guarding both. + +## Requirements + +- Functional: `make fetch-dict` downloads the pinned JSONL and verifies its SHA-256; + `make dict` builds `data/noitu.db` from it. +- Functional: the Docker `dict` stage does the same, and its fixture escape hatch + (`FIXTURE_DICT=1`) keeps working untouched. +- Functional: the fetched source file is gitignored — we distribute the derived database, + never a copy of their file. +- Non-functional: `DICT_URL` / `DICT_SHA256` keep their names so + `web/tests/dictionary-source.test.js` continues to guard Makefile/Dockerfile agreement + with no change to the test. + +## Architecture + +The pin is a commit-addressed raw URL, which is immutable — unlike `master`, where the +URL and the checksum could silently diverge: + +```make +DICT_URL := https://raw.githubusercontent.com/undertheseanlp/dictionary/2c078cfc373b06e2980d324ce1d7bd13740c3319/dictionary/words.txt +DICT_SHA256 := 4c3e0e6117e4bdfa97731e135c3d4a05881889909267394a8de8d88ef79f13f0 +DICT_SRC := data/undertheseanlp-words.jsonl +``` + +`fetch-dict` and `verify-dict` keep their current shape — `curl` then `sha256sum -c` — +because that is already the right shape; only the size in the help text changes. + +## Related Code Files + +- Modify: `Makefile` — `DICT_URL`, `DICT_SHA256`, `DICT_SRC`, the `dict` target's flags + (`--merged` in place of `--in`), and the `help` text's "~179 MB" +- Modify: `Dockerfile` — `ARG DICT_URL`, `ARG DICT_SHA256`, the `curl`/`sha256sum` lines, + the `build-dictionary` invocation, and the `FIXTURE_DICT` comment's 179 MB reference +- Modify: `.gitignore` — ignore `data/undertheseanlp-words.jsonl` (`data/*.db` already + covers the derived and old source databases) +- Verify unchanged: `web/tests/dictionary-source.test.js`, `.github/workflows/ci.yml` + +## Implementation Steps + +1. Update the three Makefile variables and the `dict` target to pass `--merged $(DICT_SRC)`. +2. Reword `help`, `fetch-dict` and the `$(DICT_SRC)` guard: the download is ~4.8 MB now, + and saying "179 MB" would be the kind of stale comment that outlives three refactors. +3. Mirror all of it in the Dockerfile stage. Keep `FIXTURE_DICT=1` on `--words` — the + image smoke test must not start needing a network. +4. Add the source file to `.gitignore` and confirm `git status` is clean after a fetch. +5. Run `make fetch-dict && make verify-dict && make dict` end to end from a clean `data/`. +6. Run `npm test` in `web/` — `dictionary-source.test.js` should pass untouched. If it + needs editing, the variable names were changed unnecessarily; put them back. +7. Build the image with and without `FIXTURE_DICT=1` and confirm both produce a database. + +## Success Criteria + +- [ ] `make fetch-dict` pulls 4,813,111 bytes and the checksum verifies. +- [ ] `make dict` produces `data/noitu.db` with the Phase 1 word count. +- [ ] `web/tests/dictionary-source.test.js` passes with no edits to it. +- [ ] `docker build` succeeds both with `FIXTURE_DICT=1` and without. +- [ ] `git status` is clean after a fetch — no source file staged, ever. +- [ ] No file in the repository still claims a 179 MB dictionary download. + +## Risk Assessment + +**`raw.githubusercontent.com` is not an archive.** An 8-year-dormant repository could be +deleted or renamed, and then `make fetch-dict` fails for everyone. Signal: a 404 on fetch. +Response: the checksum makes any mirror verifiable, so host a copy as a release asset on +this repository and re-pin to it. Worth doing pre-emptively if the build ever gates CI. + +**The Docker stage and the Makefile drift.** Signal: the agreement test fails. Response: +that is the test doing its job — fix the file that is behind, and do not weaken the test to +match. diff --git a/plans/260908-1525-dictionary-corpus-switch/phase-03-audit-the-corpus.md b/plans/260908-1525-dictionary-corpus-switch/phase-03-audit-the-corpus.md new file mode 100644 index 0000000..327e98f --- /dev/null +++ b/plans/260908-1525-dictionary-corpus-switch/phase-03-audit-the-corpus.md @@ -0,0 +1,92 @@ +--- +phase: 3 +title: "Phase 3: Audit the corpus" +status: todo +priority: P1 +effort: "2h" +dependencies: [1, 2] +--- + +# Phase 3: Audit the corpus + +## Overview + +The gate. A rule that removes thousands of words on a capitalization heuristic has to be +looked at before it ships. This phase samples what was dropped, measures what the corpus +gained and lost, and confirms the game graph is not worse. + +## Requirements + +- Functional: a hand audit of 100 randomly sampled proper-noun drops, recorded in the + plan directory with the verdict per entry. +- Functional: a measured diff against the current `data/noitu.db` — gained, lost, and the + reason for each loss bucket. +- Functional: a playability comparison on the graph the game actually walks. +- Non-functional: every number in this phase is reproducible from a command written down + in the audit note, not from a one-off shell line nobody kept. + +## Architecture + +Three measurements, each against the freshly built database and the current one: + +1. **Drop audit.** Sample 100 of the dropped keys. Each is *proper noun* (correct), + *common word* (a false positive, the thing we are looking for), or *unclear*. The rule + passes at ≤2 common words in 100; that tolerance is the difference between a filter and + a corpus edit. +2. **Corpus diff.** Overlap, gained, lost. Split the lost into: dropped as proper nouns, + `tudientv`-only, absent from undertheseanlp. The report measured 4,145 / 1,300 / 6,473 + under all-source case evidence; this re-measures under the shipped rule. +3. **Playability.** Words, syllables, syllables that can open a word, how many have ≥2 + continuations, and dead-end syllables — the last of these being what decides whether + the game hands somebody an unanswerable position. Current: 48,216 / 6,676 / 5,049 / + 3,682 / 1,627. + +## Related Code Files + +- Create: `plans/260908-1525-dictionary-corpus-switch/audit-proper-noun-drops.md` — + the sample, the verdicts, the commands that produced them +- Read only: `data/noitu.db` (current, before regeneration), the newly built database + +## Implementation Steps + +1. Build the new database to a scratch path, leaving the shipped one in place for + comparison. Do not overwrite `data/noitu.db` in this phase. +2. Emit the dropped keys from the builder — a `--report-drops ` flag, or a temporary + local patch if the flag would otherwise have no user. Prefer the flag: the next person + to change this rule will want it too. +3. Sample 100 with a fixed seed, write them into the audit note, and judge each one by + hand. Vietnamese place and person names are the expected content; anything that reads + as an ordinary word is a false positive and gets called one. +4. Run the corpus diff and the playability comparison; put both tables in the audit note. +5. Measure the documented cost of allowed-sources-only case evidence: how many drops have + a lowercase form only in `tudientv`. +6. Decide: pass, narrow, or abandon the rule. Record the decision and its reason in the + audit note — and if the rule is narrowed, Phase 1's tests change with it. + +## Success Criteria + +- [ ] The audit note exists, with 100 judged entries and the commands that produced them. +- [ ] ≤2 of 100 sampled drops are ordinary words. +- [ ] The new corpus has >55,000 words. +- [ ] Syllables ≥ 6,676 and dead-end syllables ≤ 1,627 — the graph is not worse than + what players walk today. +- [ ] The three loss buckets are quantified, not estimated. +- [ ] A pass/narrow/abandon decision is written down with its reason. + +## Risk Assessment + +**The audit passes on a sample and the corpus is still wrong in the tail.** 100 of several +thousand is a sample, not a proof. Signal: players disputing rejected words after release. +Response: the drop list is a file — a disputed word can be checked against it in seconds, +and a per-word exception list is a small change on top of this design. + +**The audit fails and the phase becomes a redesign.** Signal: >2 common words in 100. +Response: apply Phase 1's pre-decided narrowing (every syllable capitalized), re-audit +once. If it fails again, ship without the drop — the corpus is still bigger and denser +than today's, and the proper-noun problem stays exactly as bad as it currently is rather +than getting worse. + +**Playability regresses in a way these five numbers do not capture.** Signal: aggregate +counts look fine but bot games end oddly short or long. Response: `server/internal/bot`'s +real-corpus tests run bot-versus-bot games on the shipped database; run them before +accepting the phase, since they exercise the graph rather than counting it. diff --git a/plans/260908-1525-dictionary-corpus-switch/phase-04-relicense-the-data.md b/plans/260908-1525-dictionary-corpus-switch/phase-04-relicense-the-data.md new file mode 100644 index 0000000..472d02d --- /dev/null +++ b/plans/260908-1525-dictionary-corpus-switch/phase-04-relicense-the-data.md @@ -0,0 +1,110 @@ +--- +phase: 4 +title: "Phase 4: Relicense the data" +status: todo +priority: P1 +effort: "2h" +dependencies: [3] +--- + +# Phase 4: Relicense the data + +## Overview + +`data/noitu.db` becomes GPLv3, because `hongocduc`'s data is GPL and GPL is copyleft. This +phase rewrites the data half of the licensing documents and regenerates the shipped +database — in one commit, because a GPL-derived database sitting in a tree that still +says CC BY-SA 4.0 is the only ordering mistake here with a legal consequence. + +## Requirements + +- Functional: `data/LICENSE` carries the GPLv3 text. +- Functional: `NOTICE` section 2 describes GPLv3 for the data, names both source branches + with their own licenses, and keeps section 1 (Apache-2.0 for code) intact. +- Functional: `data/ATTRIBUTION.md` records the new source, the per-branch licenses, the + full list of modifications, and why `tudientv` is excluded. +- Functional: the regenerated `data/noitu.db` carries matching `meta` provenance. +- Non-functional: no overclaiming. The `hongocduc` GPL statement comes from a mirror's + README and the document says so. + +## Architecture + +The dual-license structure already in `NOTICE` is correct and stays; only its second +section changes: + +``` +1. SOURCE CODE — Apache-2.0 (unchanged: server/, tools/, web/, proto/) +2. DICTIONARY DATA — GPLv3 (was CC BY-SA 4.0) + data/noitu.db, data/LICENSE, data/ATTRIBUTION.md +``` + +Why GPLv3 rather than CC BY-SA 4.0 is worth stating in `ATTRIBUTION.md` rather than left +implicit: `hongocduc` is Hồ Ngọc Đức's FVDP wordlist, distributed under GNU GPL. The +previous source aggregated that same data and redistributed it as CC BY-SA 4.0, which GPL +does not permit. Ours is the stricter, and correct, direction. + +The modifications list gains two entries beyond the current five — source selection and +the proper-noun drop — both of which the license requires us to declare as changes. + +## Related Code Files + +- Modify: `data/LICENSE` — CC BY-SA 4.0 text replaced with GPLv3 +- Modify: `NOTICE` — section 2 rewritten +- Modify: `data/ATTRIBUTION.md` — source table, per-branch licenses, modifications 6 and 7, + the `tudientv` exclusion note +- Modify: `README.md` — any dictionary-source or data-license claim +- Verify: `.github/workflows/ci.yml` — the required-files check (`data/LICENSE`, + `data/ATTRIBUTION.md`, `NOTICE`, `data/noitu.db`) still holds +- Regenerate: `data/noitu.db` + +## Implementation Steps + +1. Replace `data/LICENSE` with the GPLv3 text, verbatim and complete. +2. Rewrite `NOTICE` section 2: GPLv3, the affected artifacts, both upstream branches with + their licenses, and the copyleft obligation carrying into container images — the same + point the current text makes about share-alike, which is no less true of GPL. +3. Rewrite `data/ATTRIBUTION.md`: + - source is `undertheseanlp/dictionary` at commit `2c078cf`, file `dictionary/words.txt` + - `hongocduc` (GNU GPL, per the branch README — a mirror of Hồ Ngọc Đức's 2003 + wordlist, whose canonical site no longer resolves) and `wiktionary` (CC BY-SA, a + 2018-12-10 scrape of vi.wiktionary.org) + - `tudientv` **excluded**: its README declares copyright *"Chưa rõ"* and names + Soha/Vietlex (Hoàng Phê) as its primary source. Say it plainly so nobody reopens it. + - modifications 1–5 carried over, plus **6. source selection** and **7. proper-noun + removal** with the count from Phase 3 + - the resulting license and why it is GPLv3 and not CC BY-SA 4.0 +4. Grep the repository for `minhqnd`, `CC BY-SA`, `179 MB` and fix every survivor outside + `plans/` — the reports are dated records and stay as written. +5. Regenerate `data/noitu.db` and confirm its `meta` rows agree with `ATTRIBUTION.md`. A + database whose provenance contradicts the attribution file is worse than neither. +6. Run the CI required-files check locally, or read it and confirm by hand. + +## Success Criteria + +- [ ] `data/LICENSE` is the complete GPLv3 text. +- [ ] `NOTICE` section 2 says GPLv3 and names both branches with their licenses; section 1 + is unchanged. +- [ ] `data/ATTRIBUTION.md` lists seven modifications, states the `tudientv` exclusion and + its reason, and does not claim more about the `hongocduc` license than the mirror + supports. +- [ ] `meta` in the built database matches the attribution file — source URL, commit, + license, sources kept and excluded, proper-noun drop count. +- [ ] No file outside `plans/` still describes the data as CC BY-SA 4.0 or names minhqnd + as the source. +- [ ] CI's required-files check passes. + +## Risk Assessment + +**Relicensing is hard to walk back.** Once a GPLv3 `noitu.db` is published, that release +is GPLv3 permanently. Signal: none — this is simply true. Response: it is why the user +signed off before this plan was written; the mitigation is that it was a decision rather +than a side effect. + +**A stale CC BY-SA claim survives somewhere.** A README line or a Docker label saying the +wrong license is a licensing misstatement, not a typo. Signal: the grep in step 4 finds +something after the phase is called done. Response: grep is a step in the phase for that +reason; run it last, not first. + +**Section 1 gets damaged while section 2 is rewritten.** The code's Apache-2.0 grant is +not in scope and must come out byte-identical. Signal: a diff touching section 1. +Response: review the `NOTICE` diff before committing. diff --git a/plans/260908-1525-dictionary-corpus-switch/phase-05-retire-the-sqlite-path.md b/plans/260908-1525-dictionary-corpus-switch/phase-05-retire-the-sqlite-path.md new file mode 100644 index 0000000..f48e15b --- /dev/null +++ b/plans/260908-1525-dictionary-corpus-switch/phase-05-retire-the-sqlite-path.md @@ -0,0 +1,88 @@ +--- +phase: 5 +title: "Phase 5: Retire the SQLite path" +status: todo +priority: P2 +effort: "1h" +dependencies: [4] +--- + +# Phase 5: Retire the SQLite path + +## Overview + +Delete the input mode nothing uses any more: reading a source SQLite database, with the +table and column auto-detection that existed because the upstream schema was not ours to +rely on. Two input modes for one source is dead weight, and dead weight in a build tool is +where stale assumptions hide. + +## Requirements + +- Functional: `--in`, `--table`, `--word-col`, `--lang-col` and `--lang` are gone, along + with the schema resolution behind them. +- Functional: `--merged` (the corpus) and `--words` (the fixture) remain. The fixture path + stays because the e2e suite and the Docker smoke build depend on it. +- Non-functional: the package doc comment describes what the tool now does, not what it + used to. +- Non-functional: `modernc.org/sqlite` stays a dependency — it writes the output database. + +## Architecture + +What `run()` dispatches on collapses from three modes to two: + +``` +--merged the corpus: undertheseanlp JSONL (default path) +--words the fixture: one word per line (e2e, Docker smoke build) +``` + +`resolveSource` and its auto-detection tests go with `--in`. They were never about the +game's rules — they were insurance against an upstream schema we did not control, and we +no longer read that upstream. + +## Related Code Files + +- Modify: `server/cmd/build-dictionary/main.go` — remove `--in` and the four schema flags, + `resolveSource`, the SQLite read path, and the `config` fields behind them; rewrite the + package doc comment and the usage example +- Modify: `server/cmd/build-dictionary/main_test.go` — remove the source-resolution tests; + keep and extend the ones about output shape +- Modify: `Makefile` — drop any remaining `--in` reference and 179 MB wording from `help` +- Verify: `Dockerfile` no longer mentions `--in` + +## Implementation Steps + +1. Remove the flags, the `config` fields, `resolveSource` and the SQLite read path. +2. Delete the tests that exist only to cover auto-detection. Do not delete tests covering + output shape, `verify()`, or the reject-reason counts — those still describe behavior. +3. Rewrite the package doc comment: the input is a 4.8 MB JSONL wordlist with source + membership and capitalization; the output is the game's syllable-indexed database. +4. `go build ./... && go vet ./... && go test ./...` from `server/`. +5. Grep for `--in`, `dictionary.db`, `resolveSource` and `lang_code` outside `plans/`; + nothing should survive except in the dated reports. +6. `make dict` once more from a clean `data/` to prove the surviving path is the one that + works. + +## Success Criteria + +- [ ] `go build`, `go vet` and `go test ./...` are clean from `server/`. +- [ ] `build-dictionary --help` lists `--merged` and `--words` and nothing about tables, + columns or language codes. +- [ ] `make dict` and `make fixture-dict` both work. +- [ ] The e2e suite still builds its fixture and passes. +- [ ] No reference to `--in`, `resolveSource` or `lang_code` survives outside `plans/`. + +## Risk Assessment + +**Deleting the wrong tests.** The auto-detection tests and the output-shape tests live in +the same file. Signal: coverage of `verify()` or the reject counters disappears with the +deletion. Response: read `main_test.go` in full before cutting, and delete by test +function rather than by line range. + +**Somebody wants the SQLite path back.** A future source could ship SQLite again. Signal: +a future plan proposing it. Response: git remembers, and re-adding a reader for a schema +we would then actually control is a smaller job than maintaining auto-detection for a +schema nobody reads. Deleting it is the cheaper bet. + +**The fixture path is quietly coupled to something removed.** Signal: `make fixture-dict` +or the e2e build fails after the cut. Response: step 6 builds both paths; run it before +calling the phase done, not after. diff --git a/plans/260908-1525-dictionary-corpus-switch/plan.md b/plans/260908-1525-dictionary-corpus-switch/plan.md new file mode 100644 index 0000000..48a1cc7 --- /dev/null +++ b/plans/260908-1525-dictionary-corpus-switch/plan.md @@ -0,0 +1,128 @@ +--- +title: "Dictionary corpus switch" +description: "Replace the 179 MB minhqnd aggregate with undertheseanlp/dictionary (hongocduc + wiktionary, tudientv excluded), drop always-capitalized proper nouns, and relicense data/noitu.db to GPLv3" +status: pending +priority: P1 +effort: "~1d" +tags: [dictionary, data, licensing, build] +created: 2026-09-08 +blockedBy: [] +blocks: [] +--- + +# Dictionary corpus switch + +## Overview + +`data/noitu.db` is derived from `minhqnd/dictionary` v2.0.0: a 179 MB SQLite aggregate of +five upstreams, redistributed as CC BY-SA 4.0. Two problems. Its license chain does not +close — three of its five inputs are GPL (FVDP, tudientv, "Vietnamese Explanatory +Dictionary") and GPL does not permit relicensing to CC BY-SA. And it has already +lowercased everything (0 of 70,511 Vietnamese rows carry uppercase), which destroys the +one signal that separates a place name from a word. + +This plan switches the corpus to `undertheseanlp/dictionary` — a single 4.8 MB JSONL file +carrying, per word, its raw capitalization and which of three source dictionaries contain +it. `hongocduc` (GPL) and `wiktionary` (CC BY-SA) are taken; `tudientv` is excluded +permanently because its own README declares copyright *"Chưa rõ"* and names Soha/Vietlex +(Hoàng Phê) as its primary source. Entries that appear only ever capitalized are dropped +as proper nouns. `data/noitu.db` then ships **GPLv3** — the honest license for +FVDP-derived data, and the reason this is a licensing improvement rather than a new risk. + +Evidence, all measured through the real `build-dictionary` filter: +`plans/reports/research-260908-1507-undertheseanlp-dictionary.md`. + +## What changes, in numbers + +| | current | after | +|---|---|---| +| words | 48,216 | **~61,276** | +| syllables | 6,676 | 7,011 | +| syllables that can open a word | 5,049 | 5,496 | +| …with ≥2 continuations | 3,682 | 4,168 | +| dead-end syllables | 1,627 | **1,515** | +| source download | 179 MB SQLite | 4.8 MB JSONL | +| data license | CC BY-SA 4.0 | **GPLv3** | + +24,978 words gained, 11,918 lost — of which 4,145 are proper nouns removed on purpose, +1,300 are `tudientv`-only, and 6,473 are absent from undertheseanlp entirely, including +genuine modern vocabulary (`tế bào gốc`, `hằng số vũ trụ`) that a 2003 wordlist cannot +have. That loss is accepted here and is the subject of a separate future decision, not +this plan. + +## Decisions taken + +- **`tudientv` is never read.** Not as content, and not as case evidence. Excluding it + from evidence too is what makes "we do not touch that data" true without qualification. + It costs a little accuracy in the proper-noun rule — a word capitalized in `hongocduc` + but lowercase only in `tudientv` will be dropped — and Phase 3 measures that cost. +- **The measured 61,276 used all-source case evidence.** Under the allowed-sources-only + rule the number will shift slightly. Phase 1 re-measures; the plan does not depend on + the exact figure, only on it staying above the 40,000 `--min-words` floor and above + today's 48,216. +- **Proper nouns are dropped, not tiered.** The per-word `source` array looked like a + confidence signal and is not one: `hà nội` has 2 votes, `cẩm xá` 2, `kháng đón` 2, while + `cánh diều` has 1 and `mặt trời` 2. Capitalization is the only usable signal in this + data, so it is the only one used. +- **`noitu.db` becomes GPLv3; the code stays Apache-2.0.** The existing dual-license split + in `NOTICE` holds; only its data half is rewritten. +- **The SQLite input path is deleted, not kept "just in case."** Two input modes for one + source is dead weight; the fixture path (`--words`) stays because the e2e suite and the + Docker smoke build use it. +- **The viwiktionary extractor is out of scope.** It remains the answer if a CC BY-SA + corpus is ever needed again — see `research-260908-1434-viwiktionary-as-dictionary-source.md`. + +## The pin + +``` +URL https://raw.githubusercontent.com/undertheseanlp/dictionary/2c078cfc373b06e2980d324ce1d7bd13740c3319/dictionary/words.txt +SHA256 4c3e0e6117e4bdfa97731e135c3d4a05881889909267394a8de8d88ef79f13f0 +Size 4,813,111 bytes — 79,226 JSONL rows +Commit 2c078cf (2018-12-10, the repository's last) +``` + +Pinned by commit rather than branch: `raw.githubusercontent.com///` is +immutable, so the URL and the checksum cannot disagree later. The variable names +`DICT_URL` / `DICT_SHA256` are kept so `web/tests/dictionary-source.test.js` needs no +change — only their values move. + +## Phases + +| # | Phase | Status | +|---|-------|--------| +| 1 | [Read the merged list](./phase-01-start.md) | Pending | +| 2 | [Pin the source](./phase-02-pin-the-source.md) | Pending | +| 3 | [Audit the corpus](./phase-03-audit-the-corpus.md) | Pending | +| 4 | [Relicense the data](./phase-04-relicense-the-data.md) | Pending | +| 5 | [Retire the SQLite path](./phase-05-retire-the-sqlite-path.md) | Pending | + +Phase 3 is a gate: it may send Phase 1 back to narrow the proper-noun rule. Phase 4 must +land in the same commit as the first generated GPL-derived `noitu.db` — shipping that +database while `NOTICE` still says CC BY-SA 4.0 is the one ordering mistake with a legal +consequence. + +## Success criteria + +- [ ] `make fetch-dict && make dict` builds `data/noitu.db` from a 4.8 MB pinned file with + no 179 MB download. +- [ ] The shipped database has **more than 55,000 words** and **no word that reaches it + only via `tudientv`**. +- [ ] The proper-noun drop count is reported in the build log beside the other reject + reasons, recorded in `meta`, and a 100-entry sample has been audited by hand. +- [ ] Playability does not regress: syllables ≥ 6,676 and dead-end syllables ≤ 1,627. +- [ ] `data/LICENSE`, `NOTICE` and `data/ATTRIBUTION.md` describe GPLv3, name both source + branches with their own licenses, and record why `tudientv` is excluded. +- [ ] `make test-go`, `make test-web` and the e2e suite are green; the fixture build path + is untouched. +- [ ] `--in` and its schema auto-detection are gone, and no doc still mentions the 179 MB + download. + +## Open questions + +- The `hongocduc` GPL claim rests on the mirror's README; Hồ Ngọc Đức's canonical site + 404s. Assuming GPL is the conservative direction, so this does not block — but the + evidence is a mirror, and `ATTRIBUTION.md` should say so rather than overclaim. +- The 6,473 words undertheseanlp does not have are an unquantified mix of junk and real + modern vocabulary. Separating them needs a second live source; out of scope here. + +