docs(plans): plan the switch to the undertheseanlp corpus

Five phases from the merged wordlist to a GPLv3 data artifact, with the
proper-noun audit as a gate that can send the capitalization rule back.

plans/260908-1525-dictionary-corpus-switch/phase-01-start.md

@@ -0,0 +1,120 @@
+---
+phase: 1
+title: "Phase 1: Read the merged list"
+status: todo
+priority: P1
+effort: "3h"
+dependencies: []
+---
+
+# Phase 1: Read the merged list
+
+## Overview
+
+Teach `build-dictionary` a third input: undertheseanlp's merged JSONL. It selects words by
+source membership, drops entries that appear only ever capitalized, and hands the survivors
+to the normalization and filtering path that already exists.
+
+## Requirements
+
+- Functional: read `{"text": "...", "source": ["hongocduc", ...]}` lines; keep a word when
+  its sources intersect the allowed set; drop a word when every raw form of it, across the
+  allowed sources, begins with an uppercase letter.
+- Functional: `tudientv` rows are skipped before any decision is made about a word —
+  neither its words nor its capitalization inform the output.
+- Functional: the proper-noun drop is counted and printed beside the existing reject
+  reasons, so the build log says what the source contained.
+- Non-functional: streaming line-by-line, one pass to gather, one pass to decide. The file
+  is 4.8 MB; nothing here needs to be clever.
+- Non-functional: everything downstream — `accept()`, syllable indexing, alias generation,
+  `verify()`, `meta` — is reused unchanged.
+
+## Architecture
+
+Two maps keyed by the lowercased, whitespace-collapsed word:
+
+```
+forms[key]   -> set of raw spellings seen in allowed sources   ("Mặt Trời", "mặt trời")
+sources[key] -> set of allowed sources containing it           {"hongocduc", "wiktionary"}
+```
+
+A key survives when `sources[key]` is non-empty and at least one entry in `forms[key]`
+does not start with an uppercase letter. `Mặt Trời` survives on the strength of its
+lowercase twin; `Hà Nội`, `Trương Công Định` and `A Mú Sung` have no lowercase form and go.
+
+Surviving keys are then fed to the same `accept()` the SQLite path feeds, so the ≥2
+syllable rule, the digit/punctuation rejections and the Vietnamese phonotactic check all
+apply as they do today.
+
+Case evidence deliberately comes from allowed sources only. The alternative — reading
+`tudientv` rows for their capitalization while refusing their words — would be more
+accurate and would undercut the claim that we do not use that data. Phase 3 measures what
+the stricter choice costs.
+
+## Related Code Files
+
+- Create: `server/cmd/build-dictionary/merged_list.go`
+- Create: `server/cmd/build-dictionary/merged_list_test.go`
+- Modify: `server/cmd/build-dictionary/main.go` — `--merged` and `--sources` flags, dispatch
+  in `run()`, `sourceURL`/`sourceLicense` constants, `meta` rows, package doc comment
+- Modify: `server/cmd/build-dictionary/filter.go` — add the `rejectProperNoun` reason so
+  the drop is reported through the existing counter, not a bespoke log line
+
+## Implementation Steps
+
+1. Add `rejectProperNoun rejectReason = "only ever capitalized"` to `filter.go`. Leave
+   `accept()` alone — the case decision happens before it, on evidence `accept()` cannot
+   see, since it lowercases.
+2. Write `merged_list.go`: decode with `encoding/json` per line (`bufio.Scanner` with a
+   raised buffer — the longest line is short, but a scanner that silently truncates is a
+   bad way to lose words), build the two maps, apply the rule, return words plus a count
+   per reject reason.
+3. Add `--merged <file>` and `--sources hongocduc,wiktionary` to `main.go`. Unknown source
+   names are an error, not a silent no-op: a typo in `--sources` must not quietly ship an
+   empty or wrong corpus.
+4. Dispatch in `run()`: `--merged` takes precedence in the same shape `--words` already
+   does. Refuse `--merged` together with `--in`/`--words` rather than picking one.
+5. Point `sourceURL`, `sourceLicense` and the package doc comment at the new source. Write
+   `source_commit`, `sources_kept`, `sources_excluded` and `proper_nouns_dropped` into
+   `meta` — provenance in the artifact, not only in a markdown file.
+6. Tests in `merged_list_test.go`, table-driven, over hand-written JSONL fixtures:
+   - `Mặt Trời` + `mặt trời` → kept
+   - `Hà Nội` alone → dropped as a proper noun
+   - a `tudientv`-only word → absent from the output
+   - a word whose only lowercase form is in `tudientv` → dropped (the documented cost)
+   - `học sinh` in all three → kept once, not thrice
+   - a malformed line → a clear error naming the line number, not a skipped word
+7. Run against the real file and record the counts. Compare to the report's 61,276, which
+   used all-source case evidence; explain the delta rather than adjusting the number.
+
+## Success Criteria
+
+- [ ] `go test ./cmd/build-dictionary/` passes, new tests included.
+- [ ] Building from the real merged file accepts >55,000 words and logs a proper-noun drop
+      count in the low thousands.
+- [ ] Every one of the 11 known-bad probes from the report is absent from the output
+      (`trương công định`, `trần danh án`, `tân an thạnh`, `cẩm xá`, `vĩnh điện`,
+      `cam lâm`, `hà nội`, `sài gòn`, `a mú sung`, `a lưới`, `kháng đón`).
+- [ ] All 10 ordinary-word probes are present (`học sinh`, `mặt trời`, `con người`,
+      `bánh mì`, `cánh diều`, `xe đạp`, `giáo viên`, `hoa hồng`, `tình yêu`, `nước mắm`).
+- [ ] `--sources hongocduc` alone and `--sources hongocduc,wiktionary` both build, with the
+      first strictly smaller.
+- [ ] No word in the output is reachable only through `tudientv`.
+
+## Risk Assessment
+
+**The capitalization rule eats real words.** The probe set is 21 words; the rule fires on
+thousands. Signal: Phase 3's hand audit finds common words among the drops. Response:
+narrow the rule to require every syllable capitalized (`Hà Nội`, not `Kháng đón`), re-audit,
+and if that still misfires, stop dropping and keep the count as a log line only — a corpus
+with proper nouns in it is what we ship today and is survivable.
+
+**Allowed-sources-only case evidence drops more than expected.** Signal: the drop count
+comes in far above the report's 4,145. Response: measure how many drops have a lowercase
+form only in `tudientv`; if that is a large share, reconsider reading `tudientv` for case
+evidence alone and say so plainly in `ATTRIBUTION.md` rather than hiding it.
+
+**A JSON schema surprise.** The file is 8 years old and unversioned; a row could carry an
+unexpected shape. Signal: decode errors on the real file. Response: the error names the
+line — inspect it, and only then decide between a tolerant skip with a counted reason and
+a hard failure. Silent skipping is not an option.

plans/260908-1525-dictionary-corpus-switch/phase-02-pin-the-source.md

@@ -0,0 +1,85 @@
+---
+phase: 2
+title: "Phase 2: Pin the source"
+status: todo
+priority: P1
+effort: "2h"
+dependencies: [1]
+---
+
+# Phase 2: Pin the source
+
+## Overview
+
+Point the build machinery at the new file: pinned by commit and checksum, fetched by
+`make fetch-dict` and by the Docker dictionary stage, with the existing agreement test
+still guarding both.
+
+## Requirements
+
+- Functional: `make fetch-dict` downloads the pinned JSONL and verifies its SHA-256;
+  `make dict` builds `data/noitu.db` from it.
+- Functional: the Docker `dict` stage does the same, and its fixture escape hatch
+  (`FIXTURE_DICT=1`) keeps working untouched.
+- Functional: the fetched source file is gitignored — we distribute the derived database,
+  never a copy of their file.
+- Non-functional: `DICT_URL` / `DICT_SHA256` keep their names so
+  `web/tests/dictionary-source.test.js` continues to guard Makefile/Dockerfile agreement
+  with no change to the test.
+
+## Architecture
+
+The pin is a commit-addressed raw URL, which is immutable — unlike `master`, where the
+URL and the checksum could silently diverge:
+
+```make
+DICT_URL    := https://raw.githubusercontent.com/undertheseanlp/dictionary/2c078cfc373b06e2980d324ce1d7bd13740c3319/dictionary/words.txt
+DICT_SHA256 := 4c3e0e6117e4bdfa97731e135c3d4a05881889909267394a8de8d88ef79f13f0
+DICT_SRC    := data/undertheseanlp-words.jsonl
+```
+
+`fetch-dict` and `verify-dict` keep their current shape — `curl` then `sha256sum -c` —
+because that is already the right shape; only the size in the help text changes.
+
+## Related Code Files
+
+- Modify: `Makefile` — `DICT_URL`, `DICT_SHA256`, `DICT_SRC`, the `dict` target's flags
+  (`--merged` in place of `--in`), and the `help` text's "~179 MB"
+- Modify: `Dockerfile` — `ARG DICT_URL`, `ARG DICT_SHA256`, the `curl`/`sha256sum` lines,
+  the `build-dictionary` invocation, and the `FIXTURE_DICT` comment's 179 MB reference
+- Modify: `.gitignore` — ignore `data/undertheseanlp-words.jsonl` (`data/*.db` already
+  covers the derived and old source databases)
+- Verify unchanged: `web/tests/dictionary-source.test.js`, `.github/workflows/ci.yml`
+
+## Implementation Steps
+
+1. Update the three Makefile variables and the `dict` target to pass `--merged $(DICT_SRC)`.
+2. Reword `help`, `fetch-dict` and the `$(DICT_SRC)` guard: the download is ~4.8 MB now,
+   and saying "179 MB" would be the kind of stale comment that outlives three refactors.
+3. Mirror all of it in the Dockerfile stage. Keep `FIXTURE_DICT=1` on `--words` — the
+   image smoke test must not start needing a network.
+4. Add the source file to `.gitignore` and confirm `git status` is clean after a fetch.
+5. Run `make fetch-dict && make verify-dict && make dict` end to end from a clean `data/`.
+6. Run `npm test` in `web/` — `dictionary-source.test.js` should pass untouched. If it
+   needs editing, the variable names were changed unnecessarily; put them back.
+7. Build the image with and without `FIXTURE_DICT=1` and confirm both produce a database.
+
+## Success Criteria
+
+- [ ] `make fetch-dict` pulls 4,813,111 bytes and the checksum verifies.
+- [ ] `make dict` produces `data/noitu.db` with the Phase 1 word count.
+- [ ] `web/tests/dictionary-source.test.js` passes with no edits to it.
+- [ ] `docker build` succeeds both with `FIXTURE_DICT=1` and without.
+- [ ] `git status` is clean after a fetch — no source file staged, ever.
+- [ ] No file in the repository still claims a 179 MB dictionary download.
+
+## Risk Assessment
+
+**`raw.githubusercontent.com` is not an archive.** An 8-year-dormant repository could be
+deleted or renamed, and then `make fetch-dict` fails for everyone. Signal: a 404 on fetch.
+Response: the checksum makes any mirror verifiable, so host a copy as a release asset on
+this repository and re-pin to it. Worth doing pre-emptively if the build ever gates CI.
+
+**The Docker stage and the Makefile drift.** Signal: the agreement test fails. Response:
+that is the test doing its job — fix the file that is behind, and do not weaken the test to
+match.

plans/260908-1525-dictionary-corpus-switch/phase-03-audit-the-corpus.md

@@ -0,0 +1,92 @@
+---
+phase: 3
+title: "Phase 3: Audit the corpus"
+status: todo
+priority: P1
+effort: "2h"
+dependencies: [1, 2]
+---
+
+# Phase 3: Audit the corpus
+
+## Overview
+
+The gate. A rule that removes thousands of words on a capitalization heuristic has to be
+looked at before it ships. This phase samples what was dropped, measures what the corpus
+gained and lost, and confirms the game graph is not worse.
+
+## Requirements
+
+- Functional: a hand audit of 100 randomly sampled proper-noun drops, recorded in the
+  plan directory with the verdict per entry.
+- Functional: a measured diff against the current `data/noitu.db` — gained, lost, and the
+  reason for each loss bucket.
+- Functional: a playability comparison on the graph the game actually walks.
+- Non-functional: every number in this phase is reproducible from a command written down
+  in the audit note, not from a one-off shell line nobody kept.
+
+## Architecture
+
+Three measurements, each against the freshly built database and the current one:
+
+1. **Drop audit.** Sample 100 of the dropped keys. Each is *proper noun* (correct),
+   *common word* (a false positive, the thing we are looking for), or *unclear*. The rule
+   passes at ≤2 common words in 100; that tolerance is the difference between a filter and
+   a corpus edit.
+2. **Corpus diff.** Overlap, gained, lost. Split the lost into: dropped as proper nouns,
+   `tudientv`-only, absent from undertheseanlp. The report measured 4,145 / 1,300 / 6,473
+   under all-source case evidence; this re-measures under the shipped rule.
+3. **Playability.** Words, syllables, syllables that can open a word, how many have ≥2
+   continuations, and dead-end syllables — the last of these being what decides whether
+   the game hands somebody an unanswerable position. Current: 48,216 / 6,676 / 5,049 /
+   3,682 / 1,627.
+
+## Related Code Files
+
+- Create: `plans/260908-1525-dictionary-corpus-switch/audit-proper-noun-drops.md` —
+  the sample, the verdicts, the commands that produced them
+- Read only: `data/noitu.db` (current, before regeneration), the newly built database
+
+## Implementation Steps
+
+1. Build the new database to a scratch path, leaving the shipped one in place for
+   comparison. Do not overwrite `data/noitu.db` in this phase.
+2. Emit the dropped keys from the builder — a `--report-drops <file>` flag, or a temporary
+   local patch if the flag would otherwise have no user. Prefer the flag: the next person
+   to change this rule will want it too.
+3. Sample 100 with a fixed seed, write them into the audit note, and judge each one by
+   hand. Vietnamese place and person names are the expected content; anything that reads
+   as an ordinary word is a false positive and gets called one.
+4. Run the corpus diff and the playability comparison; put both tables in the audit note.
+5. Measure the documented cost of allowed-sources-only case evidence: how many drops have
+   a lowercase form only in `tudientv`.
+6. Decide: pass, narrow, or abandon the rule. Record the decision and its reason in the
+   audit note — and if the rule is narrowed, Phase 1's tests change with it.
+
+## Success Criteria
+
+- [ ] The audit note exists, with 100 judged entries and the commands that produced them.
+- [ ] ≤2 of 100 sampled drops are ordinary words.
+- [ ] The new corpus has >55,000 words.
+- [ ] Syllables ≥ 6,676 and dead-end syllables ≤ 1,627 — the graph is not worse than
+      what players walk today.
+- [ ] The three loss buckets are quantified, not estimated.
+- [ ] A pass/narrow/abandon decision is written down with its reason.
+
+## Risk Assessment
+
+**The audit passes on a sample and the corpus is still wrong in the tail.** 100 of several
+thousand is a sample, not a proof. Signal: players disputing rejected words after release.
+Response: the drop list is a file — a disputed word can be checked against it in seconds,
+and a per-word exception list is a small change on top of this design.
+
+**The audit fails and the phase becomes a redesign.** Signal: >2 common words in 100.
+Response: apply Phase 1's pre-decided narrowing (every syllable capitalized), re-audit
+once. If it fails again, ship without the drop — the corpus is still bigger and denser
+than today's, and the proper-noun problem stays exactly as bad as it currently is rather
+than getting worse.
+
+**Playability regresses in a way these five numbers do not capture.** Signal: aggregate
+counts look fine but bot games end oddly short or long. Response: `server/internal/bot`'s
+real-corpus tests run bot-versus-bot games on the shipped database; run them before
+accepting the phase, since they exercise the graph rather than counting it.

plans/260908-1525-dictionary-corpus-switch/phase-04-relicense-the-data.md

@@ -0,0 +1,110 @@
+---
+phase: 4
+title: "Phase 4: Relicense the data"
+status: todo
+priority: P1
+effort: "2h"
+dependencies: [3]
+---
+
+# Phase 4: Relicense the data
+
+## Overview
+
+`data/noitu.db` becomes GPLv3, because `hongocduc`'s data is GPL and GPL is copyleft. This
+phase rewrites the data half of the licensing documents and regenerates the shipped
+database — in one commit, because a GPL-derived database sitting in a tree that still
+says CC BY-SA 4.0 is the only ordering mistake here with a legal consequence.
+
+## Requirements
+
+- Functional: `data/LICENSE` carries the GPLv3 text.
+- Functional: `NOTICE` section 2 describes GPLv3 for the data, names both source branches
+  with their own licenses, and keeps section 1 (Apache-2.0 for code) intact.
+- Functional: `data/ATTRIBUTION.md` records the new source, the per-branch licenses, the
+  full list of modifications, and why `tudientv` is excluded.
+- Functional: the regenerated `data/noitu.db` carries matching `meta` provenance.
+- Non-functional: no overclaiming. The `hongocduc` GPL statement comes from a mirror's
+  README and the document says so.
+
+## Architecture
+
+The dual-license structure already in `NOTICE` is correct and stays; only its second
+section changes:
+
+```
+1. SOURCE CODE          — Apache-2.0        (unchanged: server/, tools/, web/, proto/)
+2. DICTIONARY DATA      — GPLv3             (was CC BY-SA 4.0)
+                          data/noitu.db, data/LICENSE, data/ATTRIBUTION.md
+```
+
+Why GPLv3 rather than CC BY-SA 4.0 is worth stating in `ATTRIBUTION.md` rather than left
+implicit: `hongocduc` is Hồ Ngọc Đức's FVDP wordlist, distributed under GNU GPL. The
+previous source aggregated that same data and redistributed it as CC BY-SA 4.0, which GPL
+does not permit. Ours is the stricter, and correct, direction.
+
+The modifications list gains two entries beyond the current five — source selection and
+the proper-noun drop — both of which the license requires us to declare as changes.
+
+## Related Code Files
+
+- Modify: `data/LICENSE` — CC BY-SA 4.0 text replaced with GPLv3
+- Modify: `NOTICE` — section 2 rewritten
+- Modify: `data/ATTRIBUTION.md` — source table, per-branch licenses, modifications 6 and 7,
+  the `tudientv` exclusion note
+- Modify: `README.md` — any dictionary-source or data-license claim
+- Verify: `.github/workflows/ci.yml` — the required-files check (`data/LICENSE`,
+  `data/ATTRIBUTION.md`, `NOTICE`, `data/noitu.db`) still holds
+- Regenerate: `data/noitu.db`
+
+## Implementation Steps
+
+1. Replace `data/LICENSE` with the GPLv3 text, verbatim and complete.
+2. Rewrite `NOTICE` section 2: GPLv3, the affected artifacts, both upstream branches with
+   their licenses, and the copyleft obligation carrying into container images — the same
+   point the current text makes about share-alike, which is no less true of GPL.
+3. Rewrite `data/ATTRIBUTION.md`:
+   - source is `undertheseanlp/dictionary` at commit `2c078cf`, file `dictionary/words.txt`
+   - `hongocduc` (GNU GPL, per the branch README — a mirror of Hồ Ngọc Đức's 2003
+     wordlist, whose canonical site no longer resolves) and `wiktionary` (CC BY-SA, a
+     2018-12-10 scrape of vi.wiktionary.org)
+   - `tudientv` **excluded**: its README declares copyright *"Chưa rõ"* and names
+     Soha/Vietlex (Hoàng Phê) as its primary source. Say it plainly so nobody reopens it.
+   - modifications 1–5 carried over, plus **6. source selection** and **7. proper-noun
+     removal** with the count from Phase 3
+   - the resulting license and why it is GPLv3 and not CC BY-SA 4.0
+4. Grep the repository for `minhqnd`, `CC BY-SA`, `179 MB` and fix every survivor outside
+   `plans/` — the reports are dated records and stay as written.
+5. Regenerate `data/noitu.db` and confirm its `meta` rows agree with `ATTRIBUTION.md`. A
+   database whose provenance contradicts the attribution file is worse than neither.
+6. Run the CI required-files check locally, or read it and confirm by hand.
+
+## Success Criteria
+
+- [ ] `data/LICENSE` is the complete GPLv3 text.
+- [ ] `NOTICE` section 2 says GPLv3 and names both branches with their licenses; section 1
+      is unchanged.
+- [ ] `data/ATTRIBUTION.md` lists seven modifications, states the `tudientv` exclusion and
+      its reason, and does not claim more about the `hongocduc` license than the mirror
+      supports.
+- [ ] `meta` in the built database matches the attribution file — source URL, commit,
+      license, sources kept and excluded, proper-noun drop count.
+- [ ] No file outside `plans/` still describes the data as CC BY-SA 4.0 or names minhqnd
+      as the source.
+- [ ] CI's required-files check passes.
+
+## Risk Assessment
+
+**Relicensing is hard to walk back.** Once a GPLv3 `noitu.db` is published, that release
+is GPLv3 permanently. Signal: none — this is simply true. Response: it is why the user
+signed off before this plan was written; the mitigation is that it was a decision rather
+than a side effect.
+
+**A stale CC BY-SA claim survives somewhere.** A README line or a Docker label saying the
+wrong license is a licensing misstatement, not a typo. Signal: the grep in step 4 finds
+something after the phase is called done. Response: grep is a step in the phase for that
+reason; run it last, not first.
+
+**Section 1 gets damaged while section 2 is rewritten.** The code's Apache-2.0 grant is
+not in scope and must come out byte-identical. Signal: a diff touching section 1.
+Response: review the `NOTICE` diff before committing.

plans/260908-1525-dictionary-corpus-switch/phase-05-retire-the-sqlite-path.md

@@ -0,0 +1,88 @@
+---
+phase: 5
+title: "Phase 5: Retire the SQLite path"
+status: todo
+priority: P2
+effort: "1h"
+dependencies: [4]
+---
+
+# Phase 5: Retire the SQLite path
+
+## Overview
+
+Delete the input mode nothing uses any more: reading a source SQLite database, with the
+table and column auto-detection that existed because the upstream schema was not ours to
+rely on. Two input modes for one source is dead weight, and dead weight in a build tool is
+where stale assumptions hide.
+
+## Requirements
+
+- Functional: `--in`, `--table`, `--word-col`, `--lang-col` and `--lang` are gone, along
+  with the schema resolution behind them.
+- Functional: `--merged` (the corpus) and `--words` (the fixture) remain. The fixture path
+  stays because the e2e suite and the Docker smoke build depend on it.
+- Non-functional: the package doc comment describes what the tool now does, not what it
+  used to.
+- Non-functional: `modernc.org/sqlite` stays a dependency — it writes the output database.
+
+## Architecture
+
+What `run()` dispatches on collapses from three modes to two:
+
+```
+--merged <file>   the corpus:  undertheseanlp JSONL   (default path)
+--words  <file>   the fixture: one word per line      (e2e, Docker smoke build)
+```
+
+`resolveSource` and its auto-detection tests go with `--in`. They were never about the
+game's rules — they were insurance against an upstream schema we did not control, and we
+no longer read that upstream.
+
+## Related Code Files
+
+- Modify: `server/cmd/build-dictionary/main.go` — remove `--in` and the four schema flags,
+  `resolveSource`, the SQLite read path, and the `config` fields behind them; rewrite the
+  package doc comment and the usage example
+- Modify: `server/cmd/build-dictionary/main_test.go` — remove the source-resolution tests;
+  keep and extend the ones about output shape
+- Modify: `Makefile` — drop any remaining `--in` reference and 179 MB wording from `help`
+- Verify: `Dockerfile` no longer mentions `--in`
+
+## Implementation Steps
+
+1. Remove the flags, the `config` fields, `resolveSource` and the SQLite read path.
+2. Delete the tests that exist only to cover auto-detection. Do not delete tests covering
+   output shape, `verify()`, or the reject-reason counts — those still describe behavior.
+3. Rewrite the package doc comment: the input is a 4.8 MB JSONL wordlist with source
+   membership and capitalization; the output is the game's syllable-indexed database.
+4. `go build ./... && go vet ./... && go test ./...` from `server/`.
+5. Grep for `--in`, `dictionary.db`, `resolveSource` and `lang_code` outside `plans/`;
+   nothing should survive except in the dated reports.
+6. `make dict` once more from a clean `data/` to prove the surviving path is the one that
+   works.
+
+## Success Criteria
+
+- [ ] `go build`, `go vet` and `go test ./...` are clean from `server/`.
+- [ ] `build-dictionary --help` lists `--merged` and `--words` and nothing about tables,
+      columns or language codes.
+- [ ] `make dict` and `make fixture-dict` both work.
+- [ ] The e2e suite still builds its fixture and passes.
+- [ ] No reference to `--in`, `resolveSource` or `lang_code` survives outside `plans/`.
+
+## Risk Assessment
+
+**Deleting the wrong tests.** The auto-detection tests and the output-shape tests live in
+the same file. Signal: coverage of `verify()` or the reject counters disappears with the
+deletion. Response: read `main_test.go` in full before cutting, and delete by test
+function rather than by line range.
+
+**Somebody wants the SQLite path back.** A future source could ship SQLite again. Signal:
+a future plan proposing it. Response: git remembers, and re-adding a reader for a schema
+we would then actually control is a smaller job than maintaining auto-detection for a
+schema nobody reads. Deleting it is the cheaper bet.
+
+**The fixture path is quietly coupled to something removed.** Signal: `make fixture-dict`
+or the e2e build fails after the cut. Response: step 6 builds both paths; run it before
+calling the phase done, not after.

plans/260908-1525-dictionary-corpus-switch/plan.md

@@ -0,0 +1,128 @@
+---
+title: "Dictionary corpus switch"
+description: "Replace the 179 MB minhqnd aggregate with undertheseanlp/dictionary (hongocduc + wiktionary, tudientv excluded), drop always-capitalized proper nouns, and relicense data/noitu.db to GPLv3"
+status: pending
+priority: P1
+effort: "~1d"
+tags: [dictionary, data, licensing, build]
+created: 2026-09-08
+blockedBy: []
+blocks: []
+---
+
+# Dictionary corpus switch
+
+## Overview
+
+`data/noitu.db` is derived from `minhqnd/dictionary` v2.0.0: a 179 MB SQLite aggregate of
+five upstreams, redistributed as CC BY-SA 4.0. Two problems. Its license chain does not
+close — three of its five inputs are GPL (FVDP, tudientv, "Vietnamese Explanatory
+Dictionary") and GPL does not permit relicensing to CC BY-SA. And it has already
+lowercased everything (0 of 70,511 Vietnamese rows carry uppercase), which destroys the
+one signal that separates a place name from a word.
+
+This plan switches the corpus to `undertheseanlp/dictionary` — a single 4.8 MB JSONL file
+carrying, per word, its raw capitalization and which of three source dictionaries contain
+it. `hongocduc` (GPL) and `wiktionary` (CC BY-SA) are taken; `tudientv` is excluded
+permanently because its own README declares copyright *"Chưa rõ"* and names Soha/Vietlex
+(Hoàng Phê) as its primary source. Entries that appear only ever capitalized are dropped
+as proper nouns. `data/noitu.db` then ships **GPLv3** — the honest license for
+FVDP-derived data, and the reason this is a licensing improvement rather than a new risk.
+
+Evidence, all measured through the real `build-dictionary` filter:
+`plans/reports/research-260908-1507-undertheseanlp-dictionary.md`.
+
+## What changes, in numbers
+
+| | current | after |
+|---|---|---|
+| words | 48,216 | **~61,276** |
+| syllables | 6,676 | 7,011 |
+| syllables that can open a word | 5,049 | 5,496 |
+| …with ≥2 continuations | 3,682 | 4,168 |
+| dead-end syllables | 1,627 | **1,515** |
+| source download | 179 MB SQLite | 4.8 MB JSONL |
+| data license | CC BY-SA 4.0 | **GPLv3** |
+
+24,978 words gained, 11,918 lost — of which 4,145 are proper nouns removed on purpose,
+1,300 are `tudientv`-only, and 6,473 are absent from undertheseanlp entirely, including
+genuine modern vocabulary (`tế bào gốc`, `hằng số vũ trụ`) that a 2003 wordlist cannot
+have. That loss is accepted here and is the subject of a separate future decision, not
+this plan.
+
+## Decisions taken
+
+- **`tudientv` is never read.** Not as content, and not as case evidence. Excluding it
+  from evidence too is what makes "we do not touch that data" true without qualification.
+  It costs a little accuracy in the proper-noun rule — a word capitalized in `hongocduc`
+  but lowercase only in `tudientv` will be dropped — and Phase 3 measures that cost.
+- **The measured 61,276 used all-source case evidence.** Under the allowed-sources-only
+  rule the number will shift slightly. Phase 1 re-measures; the plan does not depend on
+  the exact figure, only on it staying above the 40,000 `--min-words` floor and above
+  today's 48,216.
+- **Proper nouns are dropped, not tiered.** The per-word `source` array looked like a
+  confidence signal and is not one: `hà nội` has 2 votes, `cẩm xá` 2, `kháng đón` 2, while
+  `cánh diều` has 1 and `mặt trời` 2. Capitalization is the only usable signal in this
+  data, so it is the only one used.
+- **`noitu.db` becomes GPLv3; the code stays Apache-2.0.** The existing dual-license split
+  in `NOTICE` holds; only its data half is rewritten.
+- **The SQLite input path is deleted, not kept "just in case."** Two input modes for one
+  source is dead weight; the fixture path (`--words`) stays because the e2e suite and the
+  Docker smoke build use it.
+- **The viwiktionary extractor is out of scope.** It remains the answer if a CC BY-SA
+  corpus is ever needed again — see `research-260908-1434-viwiktionary-as-dictionary-source.md`.
+
+## The pin
+
+```
+URL     https://raw.githubusercontent.com/undertheseanlp/dictionary/2c078cfc373b06e2980d324ce1d7bd13740c3319/dictionary/words.txt
+SHA256  4c3e0e6117e4bdfa97731e135c3d4a05881889909267394a8de8d88ef79f13f0
+Size    4,813,111 bytes — 79,226 JSONL rows
+Commit  2c078cf (2018-12-10, the repository's last)
+```
+
+Pinned by commit rather than branch: `raw.githubusercontent.com/<repo>/<sha>/` is
+immutable, so the URL and the checksum cannot disagree later. The variable names
+`DICT_URL` / `DICT_SHA256` are kept so `web/tests/dictionary-source.test.js` needs no
+change — only their values move.
+
+## Phases
+
+| # | Phase | Status |
+|---|-------|--------|
+| 1 | [Read the merged list](./phase-01-start.md) | Pending |
+| 2 | [Pin the source](./phase-02-pin-the-source.md) | Pending |
+| 3 | [Audit the corpus](./phase-03-audit-the-corpus.md) | Pending |
+| 4 | [Relicense the data](./phase-04-relicense-the-data.md) | Pending |
+| 5 | [Retire the SQLite path](./phase-05-retire-the-sqlite-path.md) | Pending |
+
+Phase 3 is a gate: it may send Phase 1 back to narrow the proper-noun rule. Phase 4 must
+land in the same commit as the first generated GPL-derived `noitu.db` — shipping that
+database while `NOTICE` still says CC BY-SA 4.0 is the one ordering mistake with a legal
+consequence.
+
+## Success criteria
+
+- [ ] `make fetch-dict && make dict` builds `data/noitu.db` from a 4.8 MB pinned file with
+      no 179 MB download.
+- [ ] The shipped database has **more than 55,000 words** and **no word that reaches it
+      only via `tudientv`**.
+- [ ] The proper-noun drop count is reported in the build log beside the other reject
+      reasons, recorded in `meta`, and a 100-entry sample has been audited by hand.
+- [ ] Playability does not regress: syllables ≥ 6,676 and dead-end syllables ≤ 1,627.
+- [ ] `data/LICENSE`, `NOTICE` and `data/ATTRIBUTION.md` describe GPLv3, name both source
+      branches with their own licenses, and record why `tudientv` is excluded.
+- [ ] `make test-go`, `make test-web` and the e2e suite are green; the fixture build path
+      is untouched.
+- [ ] `--in` and its schema auto-detection are gone, and no doc still mentions the 179 MB
+      download.
+
+## Open questions
+
+- The `hongocduc` GPL claim rests on the mirror's README; Hồ Ngọc Đức's canonical site
+  404s. Assuming GPL is the conservative direction, so this does not block — but the
+  evidence is a mirror, and `ATTRIBUTION.md` should say so rather than overclaim.
+- The 6,473 words undertheseanlp does not have are an unquantified mix of junk and real
+  modern vocabulary. Separating them needs a second live source; out of scope here.
+
+<!-- slug: dictionary-corpus-switch -->