From b1204e270ff4d92f3db913b13ecae873fce9b1c2 Mon Sep 17 00:00:00 2001 From: tiennm99 Date: Thu, 10 Sep 2026 09:56:31 +0700 Subject: [PATCH] docs(plans): drop delivered plan phases, keep journals and reports --- .../phase-01-foundations-and-data-pipeline.md | 146 ------- ...hase-02-go-dictionary-and-normalization.md | 150 ------- .../phase-03-go-game-engine-and-bot-ai.md | 184 --------- .../phase-04-protobuf-contract-and-codegen.md | 226 ----------- .../phase-05-go-websocket-server-and-rooms.md | 204 ---------- .../phase-06-sveltekit-frontend.md | 275 ------------- .../phase-07-online-1v1-and-release.md | 277 ------------- plans/260904-1125-noi-tu-web-game/plan.md | 381 ------------------ .../plan.md | 70 ---- plans/260907-1636-room-chat/phase-01-start.md | 118 ------ .../phase-02-room-chat-server.md | 241 ----------- .../phase-03-client-chat-panel.md | 177 -------- plans/260907-1636-room-chat/phase-04-tests.md | 146 ------- plans/260907-1636-room-chat/phase-05-docs.md | 71 ---- plans/260907-1636-room-chat/plan.md | 210 ---------- .../phase-01-engine-elimination.md | 48 --- .../phase-02-wire-contract.md | 61 --- .../phase-03-room-seats.md | 60 --- .../phase-04-client.md | 37 -- .../phase-05-tests-docs.md | 30 -- plans/260908-1006-multiplayer-rooms/plan.md | 55 --- .../audit-proper-noun-drops.md | 184 --------- .../phase-01-start.md | 118 ------ .../phase-02-pin-the-source.md | 87 ---- .../phase-03-audit-the-corpus.md | 110 ----- .../phase-04-update-the-attribution.md | 116 ------ .../phase-05-retire-the-sqlite-path.md | 96 ----- .../plan.md | 155 ------- .../measurement.md | 97 ----- .../phase-01-start.md | 118 ------ .../phase-02-fetch-the-newest-file.md | 105 ----- .../phase-03-measure-the-corpus.md | 81 ---- .../phase-04-relicense-to-4-0.md | 94 ----- .../plan.md | 142 ------- .../measurement.md | 251 ------------ .../phase-01-start.md | 192 --------- .../phase-02-fetch-the-dump.md | 104 ----- .../phase-03-meanings-in-the-database.md | 114 ------ .../phase-04-meanings-on-the-wire.md | 108 ----- .../phase-05-meanings-in-the-client.md | 133 ------ .../phase-06-measure-and-attribute.md | 122 ------ .../plan.md | 283 ------------- 42 files changed, 5977 deletions(-) delete mode 100644 plans/260904-1125-noi-tu-web-game/phase-01-foundations-and-data-pipeline.md delete mode 100644 plans/260904-1125-noi-tu-web-game/phase-02-go-dictionary-and-normalization.md delete mode 100644 plans/260904-1125-noi-tu-web-game/phase-03-go-game-engine-and-bot-ai.md delete mode 100644 plans/260904-1125-noi-tu-web-game/phase-04-protobuf-contract-and-codegen.md delete mode 100644 plans/260904-1125-noi-tu-web-game/phase-05-go-websocket-server-and-rooms.md delete mode 100644 plans/260904-1125-noi-tu-web-game/phase-06-sveltekit-frontend.md delete mode 100644 plans/260904-1125-noi-tu-web-game/phase-07-online-1v1-and-release.md delete mode 100644 plans/260904-1125-noi-tu-web-game/plan.md delete mode 100644 plans/260907-1551-room-lobby-ready-system/plan.md delete mode 100644 plans/260907-1636-room-chat/phase-01-start.md delete mode 100644 plans/260907-1636-room-chat/phase-02-room-chat-server.md delete mode 100644 plans/260907-1636-room-chat/phase-03-client-chat-panel.md delete mode 100644 plans/260907-1636-room-chat/phase-04-tests.md delete mode 100644 plans/260907-1636-room-chat/phase-05-docs.md delete mode 100644 plans/260907-1636-room-chat/plan.md delete mode 100644 plans/260908-1006-multiplayer-rooms/phase-01-engine-elimination.md delete mode 100644 plans/260908-1006-multiplayer-rooms/phase-02-wire-contract.md delete mode 100644 plans/260908-1006-multiplayer-rooms/phase-03-room-seats.md delete mode 100644 plans/260908-1006-multiplayer-rooms/phase-04-client.md delete mode 100644 plans/260908-1006-multiplayer-rooms/phase-05-tests-docs.md delete mode 100644 plans/260908-1006-multiplayer-rooms/plan.md delete mode 100644 plans/260908-1525-dictionary-corpus-switch/audit-proper-noun-drops.md delete mode 100644 plans/260908-1525-dictionary-corpus-switch/phase-01-start.md delete mode 100644 plans/260908-1525-dictionary-corpus-switch/phase-02-pin-the-source.md delete mode 100644 plans/260908-1525-dictionary-corpus-switch/phase-03-audit-the-corpus.md delete mode 100644 plans/260908-1525-dictionary-corpus-switch/phase-04-update-the-attribution.md delete mode 100644 plans/260908-1525-dictionary-corpus-switch/phase-05-retire-the-sqlite-path.md delete mode 100644 plans/260908-1525-dictionary-corpus-switch/plan.md delete mode 100644 plans/260908-1653-kaikki-viwiktionary-corpus/measurement.md delete mode 100644 plans/260908-1653-kaikki-viwiktionary-corpus/phase-01-start.md delete mode 100644 plans/260908-1653-kaikki-viwiktionary-corpus/phase-02-fetch-the-newest-file.md delete mode 100644 plans/260908-1653-kaikki-viwiktionary-corpus/phase-03-measure-the-corpus.md delete mode 100644 plans/260908-1653-kaikki-viwiktionary-corpus/phase-04-relicense-to-4-0.md delete mode 100644 plans/260908-1653-kaikki-viwiktionary-corpus/plan.md delete mode 100644 plans/260908-2056-wiktionary-dump-corpus-and-meanings/measurement.md delete mode 100644 plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-01-start.md delete mode 100644 plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-02-fetch-the-dump.md delete mode 100644 plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-03-meanings-in-the-database.md delete mode 100644 plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-04-meanings-on-the-wire.md delete mode 100644 plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-05-meanings-in-the-client.md delete mode 100644 plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-06-measure-and-attribute.md delete mode 100644 plans/260908-2056-wiktionary-dump-corpus-and-meanings/plan.md diff --git a/plans/260904-1125-noi-tu-web-game/phase-01-foundations-and-data-pipeline.md b/plans/260904-1125-noi-tu-web-game/phase-01-foundations-and-data-pipeline.md deleted file mode 100644 index 4f51c7b..0000000 --- a/plans/260904-1125-noi-tu-web-game/phase-01-foundations-and-data-pipeline.md +++ /dev/null @@ -1,146 +0,0 @@ ---- -title: "Phase 1: Foundations and Data Pipeline" -status: done -phase: 1 -priority: P1 -effort: "3d" -dependencies: [] ---- - -# Phase 1: Foundations and Data Pipeline - -## Overview - -Stand up the repo skeleton and produce `data/noitu.db` — the derived, normalized Vietnamese -wordlist of words with **at least 2 syllables** — from the upstream `minhqnd/dictionary` -SQLite release, with full CC BY-SA 4.0 compliance. Nothing downstream can be built or -tested without this data. - -## Requirements - -**Functional** -- [x] Reproducible build: upstream `dictionary.db` → `data/noitu.db`, runnable by any contributor -- [x] Output contains only Vietnamese (`lang_code = 'vi'`) entries of **2 or more space-separated syllables** -- [x] All words NFC-normalized and lowercased -- [x] Tone-placement aliases resolved (`hoà`→`hòa`, `thuý`→`thúy`, `quí`→`quý`, …) -- [x] Precomputed `first`/`last` syllable columns and out-degree table for O(1) engine lookups -- [x] Build fails loudly if entry count falls below a floor (40,000) or any row has fewer than 2 syllables -- [x] `--max-syllables` flag available (default: no cap) so a phrase cap can be applied later without a code change - -**Non-functional** -- [x] Output DB is a few MB, opens read-only, no writes at runtime -- [x] License artifacts are correct and consistent across all five locations - -## Architecture - -**Source:** [`github.com/minhqnd/dictionary`](https://github.com/minhqnd/dictionary) release -**v2.0.0** → [`dictionary.db`](https://github.com/minhqnd/dictionary/releases/download/v2.0.0/dictionary.db), -**179 MB** (verified via the GitHub API; not in git; 357k+ entries, 1,500+ language pairs). -Code MIT, **data CC BY-SA 4.0**. - -We do **not** republish a derived dictionary — every build downloads this asset and derives -locally. See `plan.md` → Data Distribution for the CI and Docker consequences. - -**Derived schema** (`data/noitu.db`): - -```sql -CREATE TABLE words ( - word TEXT PRIMARY KEY, -- normalized, e.g. "pháp luật" - first TEXT NOT NULL, -- first syllable, "pháp" - last TEXT NOT NULL, -- last syllable, "luật" - syllables INTEGER NOT NULL -- 2, 3, 4, … -) WITHOUT ROWID; -CREATE INDEX idx_words_first ON words(first); - --- Out-degree per syllable: how many words start with it. Drives bot heuristics --- and instant dead-end detection. -CREATE TABLE syllables ( - syllable TEXT PRIMARY KEY, - out_degree INTEGER NOT NULL -) WITHOUT ROWID; - --- Accepted spelling variants that map to a canonical word. Built offline so the --- runtime never has to reason about Vietnamese tone placement. -CREATE TABLE aliases ( - variant TEXT PRIMARY KEY, - canonical TEXT NOT NULL REFERENCES words(word) -) WITHOUT ROWID; - -CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT); --- rows: source_url, source_license, built_at, word_count, builder_version -``` - -**Why an alias table instead of a runtime normalizer:** Vietnamese old-style vs new-style -tone placement (`hoà` vs `hòa`) produces genuinely different codepoint sequences. A runtime -algorithm to re-place tone marks is fiddly and easy to get subtly wrong; generating both -variants once at build time is data, cheap to fix, and trivially testable. KISS. - -**Pipeline:** - -``` -dictionary.db ──► filter lang_code='vi' - ──► NFC normalize (golang.org/x/text/unicode/norm) + lowercase + collapse spaces - ──► keep len(strings.Fields(w)) >= 2 (and <= --max-syllables when set) - ──► drop entries containing digits, latin-only tokens, or punctuation - ──► dedupe - ──► generate tone-placement variants → aliases - ──► compute first/last/syllables + out_degree - ──► write noitu.db + meta - ──► assert: count >= 40000, every row >= 2 syllables, no orphan aliases -``` - -**Note on the graph:** allowing 3+ syllable words does not change the graph model — an edge -still runs `first ──word──► last`; it may simply span more syllables in between. Out-degree, -dead-end detection, and the engine's lookups are unaffected. - -## Related Code Files - -- Create: `server/cmd/build-dictionary/main.go` — CLI: `--in dictionary.db --out data/noitu.db [--max-syllables N]` -- Create: `server/cmd/build-dictionary/filter.go` — vi + ≥2-syllable + junk filters -- Create: `server/cmd/build-dictionary/aliases.go` — tone-placement variant generation -- Create: `server/cmd/build-dictionary/main_test.go` — filter/alias unit tests with fixtures -- Create: `data/LICENSE` — CC BY-SA 4.0 full text -- Create: `data/ATTRIBUTION.md` -- Create: `NOTICE` -- Create: `README.md` (replace stub) — quickstart, architecture, **License** section -- Create: `.gitignore` — `data/*.db` (covers both the 179 MB upstream and the derived DB), `web/node_modules`, `web/build`, server binaries -- Create: `Makefile` (or `Taskfile.yml`) — `make fetch-dict`, `make dict`, `make server`, `make web`, `make test` -- Create: `server/go.mod` — `module github.com/tiennm99dev/noitu/server`, Go 1.25+ (raised by modernc.org/sqlite; local toolchain is 1.26.5) -- Modify: none (repo is empty apart from `LICENSE` + `README.md`) - -## Implementation Steps - -1. `.gitignore`, `Makefile`, `server/go.mod` (`go mod init`), directory skeleton per plan layout. -2. Write `data/LICENSE` (verbatim CC BY-SA 4.0) and `data/ATTRIBUTION.md` naming: source repo + release URL, upstream sources (Wiktionary, `vntk/dictionary`), license URL, and the explicit modification list from `plan.md`. -3. Write `NOTICE` + README **License** section stating the split: Apache-2.0 code / CC BY-SA 4.0 data, and that the two are distributed as separate artifacts. -4. `server/cmd/build-dictionary`: open upstream DB read-only via `modernc.org/sqlite`, stream `vi` words. -5. Normalization: `norm.NFC`, `strings.ToLower`, `strings.Fields` → join with single space. -6. Filters: at least 2 fields (and ≤ `--max-syllables` when set); reject any token containing a digit, ASCII-only letters, or punctuation. -7. Alias generation: for each canonical word, emit old-style tone-placement variants for the `oa/oe/uy` nuclei; skip a variant if it collides with a different canonical word (log collisions). -8. Compute `first`, `last`, `syllables`, `out_degree`; write output DB in one transaction; write `meta` rows. -9. Assertions: word count ≥ 40,000; 100% of rows ≥ 2 syllables; every alias resolves; fail non-zero otherwise. -10. `make fetch-dict` downloads the 179 MB asset from the pinned release URL into `data/` (resumable, checksum-logged); `make dict` derives from it. Both documented in the README as one-time setup. -11. Unit tests on fixtures: normalization, ≥2-syllable filter (including a 3- and a 4-syllable entry accepted and a 1-syllable rejected), junk rejection, alias generation, collision handling. - -## Success Criteria - -- [x] `make fetch-dict && make dict` produces `data/noitu.db` from the upstream release -- [x] `sqlite3 data/noitu.db "SELECT COUNT(*) FROM words"` ≥ 40,000 -- [x] Zero rows where `word` has fewer than 2 syllables; 3- and 4-syllable entries present -- [x] `hoà`, `thuý`, `quí` each resolve through `aliases` to a canonical word present in `words` -- [x] `meta` records source URL, license, build timestamp, word count -- [x] `data/LICENSE`, `data/ATTRIBUTION.md`, `NOTICE`, README license section all present and mutually consistent -- [x] `go test ./cmd/...` green -- [x] `data/*.db` is git-ignored - -## Risk Assessment - -| Risk | Signal | Response | -|---|---|---| -| Upstream schema differs from README-documented shape | Query errors on first run | Inspect actual tables with `.schema` first; adapt the extraction query — the pipeline stages after extraction are schema-independent | -| Fewer than 40k clean ≥2-syllable entries | Count assertion fails | Union with [Viet74K](https://vietnamese-wordlist.duyet.net/Viet74K.txt) filtered to ≥2 syllables; add it as a second `--in` source and extend `ATTRIBUTION.md` | -| 179 MB download is slow or the release URL moves | `make fetch-dict` fails or stalls | URL is pinned to release `v2.0.0`, download is resumable, and the derived DB is cached locally so the fetch is a one-time cost per contributor | -| Allowing 3+ syllables admits multi-word phrases that are not really words | Playtest complaints | `--max-syllables` flag already present; applying a cap is a rebuild, not a code change | -| Upstream data contains proper nouns / non-words that make the game feel wrong | Playtest complaints in later phases | Add a denylist file consumed by the builder; regenerate — no code change needed | -| Alias generation creates false accepts (a variant that is a different real word) | Collision log non-empty | Collisions are skipped by design and logged; review the log before release | -| CC BY-SA obligations misread | License review | Attribution + share-alike on the data artifact only; code untouched. `NOTICE` states the boundary explicitly | diff --git a/plans/260904-1125-noi-tu-web-game/phase-02-go-dictionary-and-normalization.md b/plans/260904-1125-noi-tu-web-game/phase-02-go-dictionary-and-normalization.md deleted file mode 100644 index 829fc5d..0000000 --- a/plans/260904-1125-noi-tu-web-game/phase-02-go-dictionary-and-normalization.md +++ /dev/null @@ -1,150 +0,0 @@ ---- -title: "Phase 2: Go Dictionary and Normalization" -status: done -phase: 2 -priority: P1 -effort: "2d" -dependencies: [1] ---- - -# Phase 2: Go Dictionary and Normalization - -## Overview - -The read-side of the dictionary: a Go package that normalizes arbitrary player input the -same way the builder normalized the corpus, and a read-only SQLite store exposing the three -lookups the game engine needs. Everything above this layer treats words as opaque strings. - -## Requirements - -**Functional** -- [x] `vietnamese.Normalize(raw) (word string, syllables []string, err error)` — NFC, lowercase, whitespace collapse, syllable split (any syllable count; length rules belong to the engine) -- [x] Store resolves a player-typed variant to its canonical word via `aliases` -- [x] `Store.Resolve(word)` → canonical word + exists -- [x] `Store.WordsStartingWith(syllable)` → words (for bot move generation) -- [x] `Store.OutDegree(syllable)` → int (0 means dead end) -- [x] Store opens the DB **read-only**; concurrent-safe for many goroutines - -**Non-functional** -- [x] Lookup ≤ 1ms p99 under 100 concurrent readers -- [x] Normalization identical to the builder's — shared code path, not a reimplementation (DRY) - -## Architecture - -`internal/vietnamese` is imported by **both** `server/cmd/build-dictionary` and the server, so the -corpus and player input can never diverge. Phase 1 writes the builder against this package; -this phase hardens and tests it. - -```go -// internal/vietnamese -func Normalize(raw string) (string, []string, error) // NFC + lower + collapse + split -const MinSyllables = 2 -func HasEnoughSyllables(sylls []string) bool // len(sylls) >= MinSyllables - -// internal/dictionary — loaded fully into memory at Open; no runtime SQL. -type Store struct{ /* maps, all written once during Open */ } -func Open(path string) (*Store, error) // file:...?mode=ro, closed before Open returns -func (s *Store) Resolve(word string) (canonical string, ok bool) -func (s *Store) LastSyllable(word string) (string, bool) -func (s *Store) WordsStartingWith(syl string) []string // shared slice, do not modify -func (s *Store) OutDegree(syl string) (int, error) // ErrNotFound if unknown -func (s *Store) RandomOpeningWord(minOutDegree int) (string, error) -func (s *Store) WordCount() int -func (s *Store) AliasCount() int -func (s *Store) License() string -``` - -**Revised during implementation — measured, not assumed.** The original design queried -SQLite per lookup and kept only `syllables` in memory. Measurement rejected that: a SQL -round-trip benchmarked at **55us** against **8.9ns** for a map hit. It would also have -threatened phase 3, where the hard bot explores hundreds of candidate moves inside a -150ms budget. - -(An earlier draft justified this with a "p99 of 1.0046ms". That figure was wrong — the -Windows clock quantizes `time.Now()` to ~1ms, so it measured timer resolution, not the -code. Cite the benchmark, which does not time individual operations, and do not -reintroduce a latency test built on `time.Now()` around a nanosecond-scale call.) - -The whole dictionary is now loaded at `Open` and the database closed immediately. -Measured on the real 48,216-word corpus: **~70ms load, ~7.8 MB heap**, and per operation, -all allocation-free: - -| Operation | Cost | Allocations | -|---|---|---| -| `Resolve` | 8.9 ns | 0 | -| `WordsStartingWith` (full iteration) | 10.7 ns | 0 | -| `OutDegree` | 15 ns | 0 | -| `RandomOpeningWord` | 18 ns | 0 | - -`Resolve` returns no error (a map lookup cannot fail) and `Close` is gone — there is -nothing left open. The result is both faster and simpler: no connection pool, no prepared -statements, no tail latency. - -`WordsStartingWith` returns an `iter.Seq[string]`, not a slice. Returning the backing -slice let a caller sort, shuffle or `append` into dictionary state: verified on the real -corpus, a caller's write landed in the Store, 2,449 of 5,049 buckets had spare capacity -for `append` to scribble into, and `-race` confirmed the write/write race across -goroutines. An iterator removes the hazard structurally rather than by comment. - -`Open` validates what it loaded against the builder's `meta.word_count`, cross-checks -every `syllables.out_degree` against the words actually indexed, and rejects orphan -aliases. Without that, a truncated database opens cleanly and the server starts, rejects -every word a player types, and fails every room creation. - -**Resolve order:** exact hit in `words` → else `aliases` lookup → else not found. - -**The engine chains on the canonical word, never on the alias the player typed.** -Canonicalization can move **either end** of the word. In the shipped dictionary roughly -half the aliases differ in the last syllable and more than a third in the first — typing -`sỹ hai` resolves to `sĩ hai`, moving the first syllable from `sỹ` to `sĩ`. An engine that -link-checked against the syllables the player typed would therefore **reject legal moves**. -`Resolve` returns the canonical form and `FirstSyllable`/`LastSyllable` report that form's -ends; phase 3 must use those, never `vietnamese.Normalize`'s split of the raw input. -All lookups are map reads; the database is closed before `Open` returns. - -**Hot-path caching:** `OutDegree` is read on every bot move. Load the whole `syllables` -table into a `map[string]int` at `Open` (a few tens of thousands of entries, ~1MB) and serve -from memory. `WordsStartingWith` stays on SQL — the result sets are small and per-turn. - -**Opening word selection:** pick uniformly from words whose `last` syllable has -`out_degree >= minOutDegree`, so a game never dies on move one. - -## Related Code Files - -- Create: `server/internal/vietnamese/normalize.go` -- Create: `server/internal/vietnamese/normalize_test.go` -- Create: `server/internal/dictionary/store.go` -- Create: `server/internal/dictionary/store_test.go` -- Create: `server/internal/dictionary/testdata/mini.db` — small fixture DB built by a test helper -- Modify: `server/go.mod` — add `modernc.org/sqlite`, `golang.org/x/text` -- Modify: `server/cmd/build-dictionary/main.go` — import `internal/vietnamese` instead of local normalization - -## Implementation Steps - -1. Implement `Normalize`: `norm.NFC.String` → `strings.ToLower` → `strings.Fields` → rejoin; return error on empty input. -2. Point the phase-1 builder at `internal/vietnamese` so one normalizer serves both sides. -3. `dictionary.Open`: DSN with `mode=ro`, verify `meta` table exists and log `word_count` + `source_license` at startup (license visibility). -4. Prepared statements for `Resolve` (words), `Resolve` (aliases), `WordsStartingWith`. -5. Load `syllables` into an in-memory map at open; `OutDegree` reads the map. -6. `RandomOpeningWord` via `ORDER BY RANDOM() LIMIT 1` over a joined out-degree filter. -7. Test helper that builds a tiny fixture DB in `t.TempDir()` from a hardcoded word set — no dependency on the real 60k DB in unit tests. -8. Tests: NFC equivalence (composed vs decomposed `ữ`), uppercase input, tabs/NBSP/multiple spaces, 1-syllable rejected while 3- and 4-syllable accepted, alias resolution for `hoà`/`thuý`/`quí`, unknown word, dead-end syllable returns 0. -9. Concurrency test: 100 goroutines × 1000 `Resolve` calls, `-race` clean. - -## Success Criteria - -- [x] `go test ./internal/... -race` green -- [x] Composed and decomposed spellings of the same word both resolve to one canonical entry -- [x] Words of 2, 3, and 4 syllables all resolve; a 1-syllable input is rejected by `HasEnoughSyllables` -- [x] `hoà lợi`-style tone variants resolve via `aliases` -- [x] Store refuses to open a missing or writable-mode DB path with a clear error -- [x] Startup log line names the data source and license -- [x] Builder and server share one normalization implementation (no duplicate NFC/lowercase logic) - -## Risk Assessment - -| Risk | Signal | Response | -|---|---|---| -| `modernc.org/sqlite` read performance disappoints under load | p99 lookup > 1ms in the concurrency test | Widen the in-memory cache: load `words` into a `map[string]string` too (~60k entries, a few MB) and use SQL only for cold paths | -| Normalization drift between builder and server | A word in the DB fails to match the same string typed by a player | Prevented structurally by the shared package; a regression test asserts round-trip on a sample of real DB rows | -| `strings.Fields` mishandles an exotic Unicode space | Rare rejection reports | `strings.Fields` already splits on all `unicode.IsSpace`; test explicitly covers NBSP and tab | diff --git a/plans/260904-1125-noi-tu-web-game/phase-03-go-game-engine-and-bot-ai.md b/plans/260904-1125-noi-tu-web-game/phase-03-go-game-engine-and-bot-ai.md deleted file mode 100644 index 6896ffa..0000000 --- a/plans/260904-1125-noi-tu-web-game/phase-03-go-game-engine-and-bot-ai.md +++ /dev/null @@ -1,184 +0,0 @@ ---- -title: "Phase 3: Go Game Engine and Bot AI" -status: done -phase: 3 -priority: P1 -effort: "4d" -dependencies: [2] ---- - -# Phase 3: Go Game Engine and Bot AI - -## Overview - -Pure, transport-free game logic: rule validation, per-game state, win/loss resolution, and -the bot's move selection at three difficulties. No WebSocket, no protobuf, no timers driven -by wall clock — the engine takes a deadline as data so it stays fully unit-testable. - -## Requirements - -**Functional** -- [x] `Engine.Submit(playerID, raw)` validates: at least 2 syllables → first syllable matches current → word exists → not already used -- [x] Distinct rejection reasons, not a boolean -- [x] Used-word set per game; a word is spent for both players -- [x] Detects `no legal move remains` for the player to act -- [x] Turn deadline held as `time.Time`, enforced by the caller; engine exposes `IsExpired(now)` -- [x] Bot difficulties: Easy (random), Medium (prefer low out-degree), Hard (dead-end win + depth-limited search) -- [x] Score model: points per accepted word, chain length tracked - -**Non-functional** -- [x] Zero dependency on `wsapi` or generated protobuf types — engine speaks its own Go types -- [x] Hard bot move decision ≤ 150ms -- [x] Engine is safe under a single owning goroutine per game (documented; the room owns it) - -## Architecture - -The game is a **directed graph**: nodes = syllables, edges = words (`pháp ──"pháp luật"──► luật`). -A move traverses an unused edge from the current node. Formally this is *Directed Edge -Geography*, which is PSPACE-complete — so no perfect solver is attempted; the Hard bot uses -an instant-win check plus bounded search. Words of 3+ syllables are edges like any other — -they simply span extra syllables between `first` and `last`, so nothing in the graph model -changes. - -```go -type RejectReason int -const ( - ReasonNone RejectReason = iota - ReasonTooFewSyllables // fewer than vietnamese.MinSyllables (2) - ReasonWrongLink - ReasonNotInDictionary - ReasonAlreadyUsed - ReasonNotYourTurn - ReasonTimeout -) - -type Engine struct { - dict *dictionary.Store - used map[string]struct{} - current string // syllable the next word must start with - turn PlayerID - deadline time.Time - history []Move - scores map[PlayerID]int -} - -func New(dict *dictionary.Store, players []PlayerID, opening string, turnLimit time.Duration) (*Engine, error) -func (e *Engine) Submit(p PlayerID, raw string) (Move, RejectReason) -func (e *Engine) LegalMoves() ([]string, error) // for the player to act -func (e *Engine) HasLegalMove() (bool, error) -func (e *Engine) IsExpired(now time.Time) bool -func (e *Engine) Snapshot() State -``` - -**Validation order matters** — check cheapest and most-informative first so the player gets -the most useful message: turn → syllable count (≥2) → link → dictionary → reuse. - -**Scoring and word length:** longer words are worth more, so allowing 3+ syllables adds a -reason to reach for them rather than being merely permissive. See the scoring formula below. - -**Bot strategies** (`internal/bot`), all implementing one interface: - -```go -type Strategy interface{ Choose(e *game.Engine) (string, error) } -``` - -| Difficulty | Behaviour | -|---|---| -| **Easy** | Uniform random legal move. Adds a 400-900ms simulated "thinking" pause so it feels human | -| **Medium** | Score each legal move by the out-degree of the syllable it hands the opponent; pick randomly among the lowest quartile. Never plays a guaranteed kill | -| **Hard** | 1) if any legal move lands on a syllable with 0 remaining continuations → play it (instant win). 2) else negamax with alpha-beta, depth 4, over the remaining edge set; eval = `-log(1 + opponent legal move count)`. 3) else fall back to Medium | - -Search cost is bounded by move ordering (ascending opponent out-degree first) and a node -cap; the branching factor is the out-degree of visited syllables, typically < 50. - -**Anti-frustration:** Hard plays the instant-win move only `hardKillRate` of the time -(a tunable constant, default 0.85). A bot that always wins is not a game. - -**Scoring:** `10 + 2 × chainLength + 5 × (syllables − 2)` per accepted word, capped; final -score reported at game over. The syllable term rewards longer compounds now that they are -legal. Client persists a personal best in `localStorage` (phase 6) — no server storage. - -## Revised during implementation — measured, not assumed - -| Spec said | Now | Why | -|---|---|---| -| Validation order: turn → length → **link** → dictionary | turn → length → **dictionary** → link → reuse | Phase 2 established that 38% of aliases move the first syllable (`sỹ hai` → `sĩ hai`). The link can only be checked after resolving, or legal moves get rejected. Messages stay accurate either way | -| `dict *dictionary.Store` | `Dictionary` interface in `game` | Lets the engine and bot be tested on hand-built word graphs small enough to reason about, with no SQLite | -| `LegalMoves() ([]string, error)`, `HasLegalMove() (bool, error)` | no error returned | Phase 2's store is in-memory; these cannot fail | -| Medium **never plays a guaranteed kill** | Medium takes a win it can see | Simulation: withholding it made Medium lose to the **random** bot 97% of the time. A strategy that declines to win loses to a coin flip. Difficulty now comes from lookahead depth, not from refusing to play well | -| Eval `-log(1 + opponent move count)` | `+log(1 + moves)` at the node | Negamax evaluates from the perspective of the player *to move*, so the sign inverts. As specified, Hard searched for positions where it was about to be trapped and lost to Medium 71% of the time | -| Bot sleeps a "thinking" pause | `ThinkingDelay()` reported, caller schedules | Sleeping inside `Choose` would make every test wait in real time | -| Strategies take `*game.Engine` | `Board` interface (read-only) | A strategy can read the position but cannot play a move, so it cannot bypass `Submit` validation | - -**Measured on the real 48,216-word corpus** (60 games per pairing, alternating sides): - -| Matchup | Win rate | Mean game length | -|---|---|---| -| Hard vs Easy | 98% | 3.3 moves | -| Medium vs Easy | 87% | — | -| Hard vs Medium | **65%** | 2.9 moves | -| Easy vs Easy | — | 15.3 moves | - -Hard decision latency on the real corpus: p95 12.5ms, **max 18.6ms** against a 150ms budget. - -Sides alternate because moving first is an advantage in its own right, and fixed seating -reports that advantage as skill — on a 7-syllable graph with fixed sides the Hard-vs-Medium -figure was exactly 50%, pure first-move effect. - -**How much lookahead is worth is a property of the graph, not of the bots.** The synthetic -ladder gave 78% on one seed and 49% on another, so the synthetic test now only guards -against a gross regression (>= 45%) and the real-corpus test carries the claim. Earlier -drafts of this document quoted the synthetic 78% and a 6ms max; both were specific to a -single run and are corrected above. - -**Playability signal for phase 6 — the dominant characteristic, not an edge case.** Games -against Hard end in ~3 moves against ~15 for two random bots. The dictionary has 1,814 -dead-end syllables, so an instant kill is available at roughly 42% of opening positions and -Hard simply takes one; the search runs in only about half of its decisions. `hardKillRate` -(0.85) softens this but does not remove it. Needs playtesting before release; the levers are -a lower kill rate and dead-end-aware opening selection, both tunable constants. - -## Related Code Files - -- Create: `server/internal/game/engine.go` -- Create: `server/internal/game/state.go` — `Move`, `State`, `PlayerID`, `RejectReason` -- Create: `server/internal/game/engine_test.go` -- Create: `server/internal/bot/bot.go` — `Strategy` interface, difficulty registry, thinking delay -- Create: `server/internal/bot/strategy_easy.go` -- Create: `server/internal/bot/strategy_medium.go` -- Create: `server/internal/bot/strategy_hard.go` -- Create: `server/internal/bot/bot_test.go` -- Create: `server/internal/bot/simulate_test.go` — 100-game difficulty ladder assertion - -## Implementation Steps - -1. `state.go`: `PlayerID`, `Move{Player, Word, First, Last, Syllables, Points, At}`, `RejectReason` with `String()`, `State` snapshot struct. -2. `engine.New`: seed `used` with the opening word, set `current` to its last syllable, assign first turn. -3. `Submit`: normalize via `vietnamese.Normalize`, then run the validation order above; on success record the move, add to `used`, advance `current` and `turn`, reset deadline, add points. -4. `LegalMoves`: `dict.WordsStartingWith(current)` minus `used`. `HasLegalMove` short-circuits on the first hit rather than materializing the slice. -5. Engine unit tests, one per rejection reason, plus: reuse blocked across both players, chain advances correctly, no-legal-move detection, `IsExpired` boundary. -6. `bot.Strategy` interface + registry keyed by difficulty; shared randomized thinking delay helper. -7. Easy: uniform pick. Medium: out-degree-of-result scoring, lowest-quartile random pick, explicit guard that skips a 0-out-degree kill. -8. Hard: instant-win scan gated by `hardKillRate`; negamax + alpha-beta depth 4 with move ordering and node cap; Medium fallback. -9. `simulate_test.go`: 100 headless bot-vs-bot games per pairing; assert Hard beats Easy well above chance and Medium sits between. Deterministic via a seeded RNG. -10. Benchmark Hard's `Choose` to confirm ≤150ms on the real DB. - -## Success Criteria - -- [x] `go test ./internal/game/... ./internal/bot/... -race` green -- [x] Every `RejectReason` has a test that produces exactly it -- [x] A 1-syllable submission yields `ReasonTooFewSyllables`; 3- and 4-syllable words are accepted and score the syllable bonus -- [x] A word played by either player cannot be replayed by the other -- [x] `HasLegalMove` returns false on a synthetic dead-end board and the engine reports the correct winner -- [x] Hard win rate vs Easy > 70% over 100 seeded games; Medium strictly between Easy and Hard -- [x] `BenchmarkHardChoose` ≤ 150ms/op against the real `noitu.db` -- [x] Engine package imports no transport or protobuf package (verified by an import assertion test) - -## Risk Assessment - -| Risk | Signal | Response | -|---|---|---| -| Depth-4 search too slow on hub syllables (high out-degree) | Benchmark exceeds 150ms | Node cap already present; reduce to depth 3 for syllables with out-degree > 200 — the depth is a tunable constant | -| Hard bot feels unbeatable | Playtest win rate ≈ 0% | Lower `hardKillRate`; it exists precisely for this dial | -| Bot picks obscure words that feel unfair | Playtest complaints | Out of scope for v1 (no frequency data in the derived DB); note as a post-v1 item requiring a frequency column | -| Engine mutated from two goroutines | `-race` failures once rooms land in phase 5 | Ownership rule documented here and enforced in phase 5: exactly one goroutine per room owns its engine | diff --git a/plans/260904-1125-noi-tu-web-game/phase-04-protobuf-contract-and-codegen.md b/plans/260904-1125-noi-tu-web-game/phase-04-protobuf-contract-and-codegen.md deleted file mode 100644 index bc79ec0..0000000 --- a/plans/260904-1125-noi-tu-web-game/phase-04-protobuf-contract-and-codegen.md +++ /dev/null @@ -1,226 +0,0 @@ ---- -title: "Phase 4: Protobuf Contract and Codegen" -status: done -phase: 4 -priority: P1 -effort: "2d" -dependencies: [1] ---- - -# Phase 4: Protobuf Contract and Codegen - -## Overview - -One `.proto` file defines every message crossing the WebSocket, and `buf` generates both the -Go server types and the JavaScript client types from it. No hand-written message structs on -either side — the schema is the single source of truth for the wire contract. - -Independent of phases 2-3; can be built in parallel with them. - -## Requirements - -**Functional** -- [x] `proto/noitu/v1/game.proto` covers handshake with nickname, create/join room, start bot game, submit word, move results, turn state, game over, errors, heartbeat -- [x] `buf generate` emits Go into `server/gen/noitu/v1` and JS into `web/src/lib/proto` -- [x] Protocol version field in the handshake — `Hello.protocol_version`, `Welcome.protocol_version`, and `wsapi.ProtocolVersion = 1`. Rejecting a mismatch lands in phase 5, which is where `Hello` is first read -- [x] Generated code is committed (reviewable diffs, no toolchain needed to build) - -**Non-functional** -- [x] JS bundle cost of the protobuf runtime kept small — ESM, tree-shakeable. Measured against a budget in phase 6, where there is a bundle to measure -- [x] `buf lint` and `buf breaking` (against `main`) run in CI - -## Architecture - -**Tooling:** [`buf`](https://buf.build) drives both targets from one config. -- Go: `protoc-gen-go` (messages only — no gRPC, this is raw WS framing) -- JS: [`@bufbuild/protobuf`](https://github.com/bufbuild/protobuf-es) **v2.14.1** + - `@bufbuild/protoc-gen-es` **v2.14.1** (both verified current on npm), generated with - `target=js` plus `.d.ts`. Chosen over `protobuf.js`: it is the only fully - conformance-compliant JS implementation, ships ESM for tree-shaking, and produces a much - smaller browser bundle. - -> `buf` **is** installed (v1.69.0), retiring the plan-time note that it was missing. It stays -> optional for contributors either way: generated code is committed, so `buf` is required only -> when the schema changes. - -**Framing:** every WS frame is a binary message — `ClientMessage` client→server, -`ServerMessage` server→client, each a single `oneof`. No JSON fallback, no text frames. - -```proto -syntax = "proto3"; -package noitu.v1; -option go_package = "github.com/tiennm99dev/noitu/server/gen/noitu/v1;noituv1"; - -enum Difficulty { DIFFICULTY_UNSPECIFIED = 0; EASY = 1; MEDIUM = 2; HARD = 3; } - -enum RejectReason { - REJECT_UNSPECIFIED = 0; - TOO_FEW_SYLLABLES = 1; // fewer than 2 syllables - WRONG_LINK = 2; - NOT_IN_DICTIONARY = 3; - ALREADY_USED = 4; - NOT_YOUR_TURN = 5; - TIMEOUT = 6; -} - -enum GameEndReason { - END_UNSPECIFIED = 0; - END_TIMEOUT = 1; - END_NO_LEGAL_MOVE = 2; - END_OPPONENT_LEFT = 3; - END_RESIGNED = 4; -} - -message Hello { uint32 protocol_version = 1; string resume_token = 2; string nickname = 3; } -message StartBotGame { Difficulty difficulty = 1; } -message CreateRoom {} -message JoinRoom { string room_code = 1; } -message SubmitWord { string word = 1; uint32 turn_seq = 2; } // turn_seq guards double-submits -message Resign {} -message Ping { int64 client_time_ms = 1; } - -message ClientMessage { - oneof payload { - Hello hello = 1; StartBotGame start_bot_game = 2; CreateRoom create_room = 3; - JoinRoom join_room = 4; SubmitWord submit_word = 5; Resign resign = 6; Ping ping = 7; - } -} - -// accepted_nickname is what the server actually stored after sanitization — the client -// must display this, not the string it sent. -message Welcome { string session_id = 1; string resume_token = 2; uint32 protocol_version = 3; string accepted_nickname = 4; } -message RoomCreated { string room_code = 1; } -message RoomJoined { string room_code = 1; string opponent_name = 2; } -message PlayedWord { string word = 1; bool by_me = 2; uint32 points = 3; uint32 syllables = 4; } -message GameStarted { - string opening_word = 1; string current_syllable = 2; bool my_turn = 3; - int64 deadline_unix_ms = 4; uint32 turn_seq = 5; uint32 turn_limit_ms = 6; -} -message TurnUpdate { - PlayedWord played = 1; string current_syllable = 2; bool my_turn = 3; - int64 deadline_unix_ms = 4; uint32 turn_seq = 5; - uint32 my_score = 6; uint32 opponent_score = 7; uint32 chain_length = 8; -} -message MoveRejected { RejectReason reason = 1; string word = 2; uint32 turn_seq = 3; } -message GameOver { bool i_won = 1; GameEndReason reason = 2; uint32 my_score = 3; uint32 chain_length = 4; } -message OpponentLeft { bool can_reconnect = 1; uint32 grace_ms = 2; } -message ServerError { string code = 1; string message = 2; } // message is a UI key, not prose -message Pong { int64 client_time_ms = 1; int64 server_time_ms = 2; } - -message ServerMessage { - oneof payload { - Welcome welcome = 1; RoomCreated room_created = 2; RoomJoined room_joined = 3; - GameStarted game_started = 4; TurnUpdate turn_update = 5; MoveRejected move_rejected = 6; - GameOver game_over = 7; OpponentLeft opponent_left = 8; ServerError error = 9; Pong pong = 10; - } -} -``` - -**Design notes** -- `TurnUpdate` is sent to **both** players after every accepted move, with `by_me`/`my_turn` - rendered per recipient — the server serializes one message per player, not a broadcast. -- `deadline_unix_ms` is an absolute server timestamp; the client renders a countdown from it. - `Ping`/`Pong` carry both clocks so the client can correct for offset. Never trust the client clock. -- `turn_seq` on `SubmitWord` makes double-submits and late submissions detectable server-side. -- `ServerError.message` carries a **UI key** (e.g. `room_not_found`), not user-facing prose — - the Vietnamese strings live in the frontend so all copy stays in one place. -- `Hello.nickname` is a **request**, not a fact. The server sanitizes it (phase 5) and returns - the stored value in `Welcome.accepted_nickname`; `RoomJoined.opponent_name` is always a - server-sanitized string. A client must never render another player's raw input. -- The engine's `RejectReason` and this proto enum are deliberately separate types with an - explicit mapping function; the wire contract must not be hostage to internal refactors. - -## Related Code Files - -- Create: `proto/noitu/v1/game.proto` -- Create: `buf.yaml`, `buf.gen.yaml`, `buf.lock` -- Create: `server/gen/noitu/v1/game.pb.go` (generated, committed) -- Create: `web/src/lib/proto/game_pb.js` (generated, committed) -- Create: `server/internal/wsapi/convert.go` — engine ↔ proto enum mapping + tests -- Modify: `Makefile` — `make proto` -- Modify: `server/go.mod` — `google.golang.org/protobuf` -- Modify: `web/package.json` — `@bufbuild/protobuf`, dev dep `@bufbuild/protoc-gen-es` - -## Implementation Steps - -1. `buf.yaml` (module + lint/breaking config) and `buf.gen.yaml` with the two plugins and their output dirs. -2. Write `game.proto` as above; `buf lint` clean. -3. `make proto` → `buf generate`; commit both generated trees. -4. `convert.go`: `game.RejectReason` → `noituv1.RejectReason`, engine end reasons → `GameEndReason`. Exhaustive `switch` with a default that returns the UNSPECIFIED value **and** logs — a silently-dropped new reason is a bug. -5. Round-trip test in Go: marshal each `ServerMessage` variant, unmarshal, assert equality. -6. Round-trip test in JS (Vitest): decode a fixture emitted by the Go test, assert field values — proves cross-language wire compatibility, not just self-consistency. -7. CI step: `buf lint`, `buf breaking --against '.git#branch=main'`, and a check that regenerating produces no diff (generated code stays in sync). - -## Success Criteria - -- [x] `make proto` regenerates both targets with zero diff on a clean tree -- [x] `buf lint` clean; `buf breaking` wired into CI -- [x] Go round-trip test covers every `oneof` variant in both directions -- [x] JS decodes a Go-produced binary fixture and reads correct values -- [x] `convert.go` mapping is exhaustive; a test fails if an engine reason gains a value with no proto counterpart -- [x] No hand-written message struct exists in `server/` or `web/` - -## Risk Assessment - -| Risk | Signal | Response | -|---|---|---| -| Schema churn breaks an already-deployed client | Decode errors after a deploy | Additive-only changes, never reuse a tag, `reserved` on removals; `protocol_version` in `Hello` lets the server reject incompatible clients with a clear message instead of failing obscurely | -| `buf` unavailable in a contributor's environment | `make proto` fails locally | Generated code is committed, so building and running never requires `buf` — only changing the schema does | -| Protobuf runtime inflates the JS bundle | Bundle budget exceeded in phase 6 | `protobuf-es` is ESM/tree-shakeable and only the generated messages are imported; measure in phase 6 and drop to hand-rolled binary framing only if it genuinely fails the budget | -| Enum drift between engine and wire | A new reject reason silently arrives as UNSPECIFIED | Exhaustive-switch test plus a logged default in `convert.go` | - - -## Phase 4 Outcome (2026-09-04) - -Delivered and verified end to end. `buf lint` clean under the `STANDARD` category with no -carve-outs, regeneration byte-identical, `go test ./... -race` green across every package, -and 23 JavaScript assertions decoding bytes the Go suite produced. - -**Two deviations from the sketch above, both accepted before implementation** - -| Sketch | Shipped | Why | -|---|---|---| -| `TOO_FEW_SYLLABLES`, `EASY`, `END_TIMEOUT` | `REJECT_REASON_TOO_FEW_SYLLABLES`, `DIFFICULTY_EASY`, `GAME_END_REASON_TIMEOUT` | buf's `STANDARD` lint mandates the enum-name prefix. Keeping the short names would have meant an `ENUM_VALUE_PREFIX` exception, weakening "buf lint clean" to "clean with a carve-out". `protobuf-es` strips the prefix, so the JS side still reads `RejectReason.WRONG_LINK` | -| `web/` untouched until phase 6 | `web/` bootstrapped as a bare npm package | The phase's own JS codegen and Vitest cross-language test have nowhere to live otherwise. Protobuf + Vitest only — no SvelteKit, no routes, no config | - -**Additions beyond the sketch** - -- `PlayedWord.typed` (field 5). `game.Move` carries both the canonical spelling and the raw - input, and the UI has to show that a correction happened rather than appearing to rewrite - the player's text. -- `REJECT_REASON_GAME_OVER` (7). `game.ReasonGameOver` exists in the engine; without a wire - counterpart the exhaustiveness test fails, which is the test working. -- `wsapi.ProtocolVersion = 1`, giving `protocol_version` a defined value rather than a field - nobody sets. - -**Toolchain** - -Both code generators resolve from the repo's own manifests — a `tool` directive in -`server/go.mod` and `web/package.json` — so regenerating needs only `buf`, with no globally -installed plugin binaries. `buf.gen.yaml` sets `clean: true`, so the JavaScript wire test -lives in `web/tests/`, outside the generated tree that gets wiped on each run. - -**Cross-language checking** - -`server/internal/wsapi/wire_test.go` writes 17 binary fixtures to `proto/testdata/` under -`-update`; both suites then read those same bytes. A JS-only round trip would only have -proven that runtime self-consistent, which is exactly the failure that ships a broken client. -`.gitattributes` marks the fixtures `binary`: nine of them contain `0x0a` and no NUL, so a -checkout with `autocrlf` enabled could otherwise rewrite them into messages that no longer -decode. - -**Review found four guards that reported green while protecting nothing.** All fixed and -then negative-tested by deliberately breaking each one: - -| Defect | Fix | -|---|---| -| `buf breaking --against '.git#branch=main'` fails on every PR — a `pull_request` checkout is a detached HEAD with no local `main` | `.git#ref=origin/main` | -| `git diff --exit-code` never reports untracked files, so a plugin emitting a *new* file would pass the sync check | `git add --intent-to-add` first, in both CI and `make proto-check` | -| The oneof-coverage test compared the schema against a hand-written list of arm names, which drifts in lockstep with the table it duplicates | Coverage now derived from what each sample message actually sets, via `WhichOneof` | -| The exhaustiveness walk ended at `String() == "unknown"`, so a reason added without a `String` case shrank the range under test and still passed | `game.NumRejectReasons` / `game.NumEndReasons` sentinels, plus a test that every reason is named | - -`WireDifficulty` was also removed: it had no caller and no message in the schema carries a -`Difficulty`, so its doc comment described a capability the contract cannot express. - -**Deferred to phase 5**, where `Hello` is first read: rejecting a mismatched -`protocol_version`, and nickname sanitization behind `Welcome.accepted_nickname`. diff --git a/plans/260904-1125-noi-tu-web-game/phase-05-go-websocket-server-and-rooms.md b/plans/260904-1125-noi-tu-web-game/phase-05-go-websocket-server-and-rooms.md deleted file mode 100644 index 668c709..0000000 --- a/plans/260904-1125-noi-tu-web-game/phase-05-go-websocket-server-and-rooms.md +++ /dev/null @@ -1,204 +0,0 @@ ---- -title: "Phase 5: Go WebSocket Server and Rooms" -status: done -phase: 5 -priority: P1 -effort: "4d" -dependencies: [3, 4] ---- - -# Phase 5: Go WebSocket Server and Rooms - -## Overview - -The transport and orchestration layer: a WebSocket endpoint that speaks Protobuf frames, -a hub owning all live rooms, per-room goroutines driving the engine and turn timers, and -the bot wired in as a virtual player. After this phase the whole game is playable over -`websocat` with no frontend. - -## Requirements - -**Functional** -- [x] `GET /ws` upgrades and runs the session loop; `GET /healthz` for liveness -- [x] Bot rooms (1 human + bot) and PvP rooms (2 humans, joined by 6-char room code) -- [x] Nicknames sanitized server-side and echoed back; opponents only ever see sanitized values -- [x] Server-authoritative 20s turn timer (`NOITU_TURN_LIMIT`); expiry ends the game for the player on turn -- [x] Every accepted move fans out a per-recipient `TurnUpdate` -- [x] Bot moves scheduled on the room goroutine after a randomized thinking delay -- [x] Session resume token issued in `Welcome`, honoured on reconnect within a grace window -- [x] Serves the built frontend as static files so one binary is the whole deployment - -**Non-functional** -- [x] One goroutine owns each room's engine — no shared mutable game state (`-race` clean) -- [x] Per-connection read limit, read/write deadlines, and rate limiting on `SubmitWord` -- [x] Graceful shutdown drains rooms and tells clients why - -## Architecture - -``` - ┌──────── hub (mutex-guarded maps) ────────┐ - conn ──► session ──chan──► room goroutine ──► game.Engine - │ │ └► bot.Strategy - │ └── time.Timer (turn deadline) - └── writer goroutine (owns the socket write side) -``` - -**Concurrency contract — the core invariant of this phase:** -- Each connection has exactly **one reader goroutine** and **one writer goroutine**. Nothing - else touches the socket. (`coder/websocket` tolerates concurrent writes, but a single - writer keeps ordering deterministic.) -- Each room has exactly **one goroutine** that owns its `game.Engine`. All input arrives as - messages on the room's channel. The engine is never locked because it is never shared. -- The hub owns only the `code → *room` and `sessionID → *session` maps, guarded by a mutex. - It never touches engine state. - -**Room lifecycle** - -``` -CreateRoom → generate code (6 chars, unambiguous alphabet: no 0/O/1/I) → room waits -JoinRoom → second player attaches → engine created → GameStarted to both -StartBotGame→ room created immediately with a bot as player 2 → GameStarted -in play → SubmitWord | timer fire | disconnect -GameOver → both notified → room lingers briefly for a rematch, then hub evicts it -``` - -**Turn timer:** the room goroutine holds a `time.Timer` for the current deadline, selected -on alongside the input channel. On fire: mark the player on turn as the loser -(`END_TIMEOUT`) and broadcast `GameOver`. The absolute deadline also goes to clients in -`TurnUpdate.deadline_unix_ms` — clients render a countdown but never decide the outcome. - -**Bot integration:** the bot is a player ID with no connection. When it is the bot's turn, -the room schedules `Choose` on a worker and delivers the result back through the same input -channel as a human move, so it flows through identical validation. If the bot has no legal -move, the human wins. - -**Reconnect:** `Welcome` carries a `resume_token`. On disconnect, the room keeps the seat -open for `graceMs` (default 30s) and sends the opponent `OpponentLeft{can_reconnect}`. A -`Hello` carrying a valid token rebinds the seat and replays current state via `GameStarted` -+ the latest `TurnUpdate`. Timeout ends the game as `END_OPPONENT_LEFT`. - -**Security / abuse limits** -- `conn.SetReadLimit(4096)` — no legitimate message approaches this (verified API: `SetReadLimit(n int64)`) -- Read deadlines come from `context.WithTimeout` passed to `conn.Read` — `coder/websocket` - has **no** `SetReadDeadline`; everything is context-driven. Ping via `conn.Ping(ctx)` every - 20s, treating a ctx timeout as a miss and closing after 2 -- `SubmitWord` rate-limited to ~5/sec per session (a submit is one dictionary lookup) -- Room codes from `crypto/rand`; join attempts rate-limited to make brute-forcing pointless -- Origin check via `websocket.AcceptOptions.OriginPatterns` from `NOITU_ALLOWED_ORIGINS`; - never set `InsecureSkipVerify` outside local dev -- **Nickname sanitization** (`sanitizeNickname`): NFC normalize, strip control characters and - zero-width joiners, collapse whitespace, trim, cap at 20 runes, reject empty → assign - `Người chơi N`. Nicknames are shown to strangers, so this is a real input-validation - boundary, not cosmetic. A denylist hook is left in place for post-v1 if abuse appears -- Every rejection returns a UI key, never a raw internal error string - -## Related Code Files - -- Create: `server/cmd/noitu-server/main.go` — flags/env, dict open, hub, HTTP mux, graceful shutdown -- Create: `server/internal/wsapi/server.go` — upgrade handler, origin check, static file serving -- Create: `server/internal/wsapi/session.go` — reader/writer goroutines, codec, resume tokens -- Create: `server/internal/wsapi/hub.go` — room + session registries, room code generation -- Create: `server/internal/wsapi/room.go` — room goroutine, timer, bot scheduling, fan-out -- Create: `server/internal/wsapi/codec.go` — protobuf marshal/unmarshal over binary frames -- Create: `server/internal/wsapi/ratelimit.go` -- Create: `server/internal/wsapi/nickname.go` — `sanitizeNickname` + tests -- Create: `server/internal/wsapi/room_test.go`, `session_test.go`, `hub_test.go` -- Modify: `server/go.mod` — `github.com/coder/websocket` -- Modify: `Makefile` — `make server`, `make run` - -## Implementation Steps - -1. `main.go`: config from env (`NOITU_ADDR`, `NOITU_DB_PATH`, `NOITU_TURN_LIMIT` default `20s`, `NOITU_ALLOWED_ORIGINS`, `NOITU_WEB_DIR`), open the dictionary read-only, construct hub, mount `/ws`, `/healthz`, and static assets, `signal.NotifyContext` shutdown. **Log the dictionary's `WordCount()`, `AliasCount()` and `License()` at startup** — the store deliberately does not log (a library writing to the global logger fights phase 5's structured logging), so the CC BY-SA attribution surfaces here or nowhere. -2. `codec.go`: `Encode(*ServerMessage) []byte` / `Decode([]byte) (*ClientMessage, error)`; binary message type only, reject text frames. -3. `session.go`: reader goroutine (read limit, per-read `context.WithTimeout`, decode, forward to the room or hub) and writer goroutine (buffered channel, single owner of writes). Generate `session_id` and `resume_token` with `crypto/rand`; sanitize `Hello.nickname`; send `Welcome` carrying `accepted_nickname`; reject mismatched `protocol_version`. -4. `hub.go`: registries + mutex; `CreateBotRoom`, `CreateRoom`, `JoinRoom(code)`, `ResumeSession(token)`, eviction of finished/idle rooms via a janitor ticker. -5. `room.go`: the goroutine — `select` over input channel, turn timer, and context done. Handle `SubmitWord` (validate turn_seq, call engine, fan out per-recipient `TurnUpdate` or `MoveRejected`), `Resign`, disconnect, timer fire, bot scheduling, and no-legal-move detection after every move. -6. Per-recipient rendering: build `TurnUpdate` twice, flipping `by_me`/`my_turn` and swapping scores. Do not broadcast one shared message. -7. Bot wiring: on bot turn, run `Choose` on a worker goroutine with the room's context, deliver via the input channel after the thinking delay; cancel cleanly if the room ends first. -8. Reconnect: grace timer per seat, `OpponentLeft` to the other player, state replay on successful resume, `END_OPPONENT_LEFT` on grace expiry. -9. `ratelimit.go`: token bucket per session for `SubmitWord` and per IP for room joins. -10. Tests with in-process WS clients: full bot game to completion; two clients play a PvP game; timeout ends the game; reuse and wrong-link rejections reach the client with the right enum; disconnect+resume within grace restores state; resume after grace fails cleanly. All under `-race`. -11. Manual smoke via `websocat` documented in the README. - -## Success Criteria - -- [x] `go test ./internal/wsapi/... -race` green -- [x] A full bot game is playable end to end over a raw WS client, no frontend involved -- [x] Two in-process clients complete a PvP game with correct alternating turns and a correct winner -- [x] Turn timeout ends the game server-side even if the client sends nothing -- [x] Disconnect + resume within 30s restores the game; after 30s the opponent wins by `END_OPPONENT_LEFT` -- [x] Read limit, rate limit, and origin check each covered by a test -- [x] `sanitizeNickname` tested against: over-length, control chars, zero-width chars, whitespace-only, and empty input; opponents never receive an unsanitized string -- [x] `go build ./cmd/noitu-server` yields one binary that serves both `/ws` and the static frontend -- [x] Graceful shutdown notifies connected clients rather than dropping sockets silently - -## Risk Assessment - -| Risk | Signal | Response | -|---|---|---| -| Engine touched from two goroutines | `-race` failure | The one-goroutine-per-room rule is structural: the engine is only reachable through the room's channel. A review checklist item, not a runtime guard | -| Timer and message race (move accepted exactly at the deadline) | Flaky test, disputed loss | Resolve entirely inside the room goroutine: the timer fire and the move arrive on the same `select`, so one strictly precedes the other. `turn_seq` makes the outcome explicit to the client | -| Bot goroutine outlives its room | Goroutine leak under load test | Room context cancels the worker; a leak test asserts goroutine count returns to baseline after N games | -| Room code collision | Join lands in the wrong room | Generate with `crypto/rand` and retry on collision inside the hub mutex | -| Restart drops all live games | Players lose in-progress matches on deploy | Accepted for v1 (documented in `plan.md`); graceful shutdown sends a typed error so the UI can explain it | -| Offensive nicknames shown to strangers | Abuse reports | Sanitization caps length and strips control/zero-width characters; a denylist hook is already in `nickname.go` for a data-only fix if it becomes a real problem | -| Reconnect replay diverges from actual state | Resumed client shows a stale board | Replay is a fresh snapshot from the engine, never a stored copy of past messages | - - -## Phase 5 Outcome (2026-09-05) - -Delivered. `go vet` and `go test ./... -race` green across every package; the whole game -is playable over a raw WebSocket with no frontend, and the binary was run against a -fixture dictionary to confirm startup, `/healthz`, the handshake, ping/pong, room -creation and the error keys end to end. - -**Two deviations from the spec above, both accepted before implementation** - -| Spec | Shipped | Why | -|---|---|---| -| Bot worker searches the live engine | Worker gets an immutable `frozenBoard` — legal moves, a copy of the used set, and the read-only dictionary | `bot.Board.Used` reads engine state, so handing a worker the live engine races every resign and disconnect the room processes while the bot thinks | -| Per-read `context.WithTimeout` on `conn.Read` | No read deadline; liveness is `conn.Ping` every 20s, 10s timeout, close after 2 misses | A read deadline cannot tell a healthy player idling in the lobby from a dead socket, and would disconnect the first | - -`NewServer` takes a `Dictionary` interface (the engine's contract plus -`RandomOpeningWord`) rather than `*dictionary.Store`, so tests play whole games against a -four-word graph whose outcome a reader can verify by eye. Tests live in `wsapi_test.go` -and `regression_test.go` rather than the three files sketched above. - -**Bugs found by writing the tests**, before review: - -| Defect | Consequence | -|---|---| -| Resume token unregistered the instant the socket dropped | A reconnect could never find its seat — the feature was entirely dead | -| `writeLoop` took its write deadline from the cancelled session context | Every refusal queued just before a close was silently discarded | -| `keyedLimiter.sweep` required a full bucket, but tokens refill lazily | Nothing was ever swept: unbounded map growth keyed by whoever connects | - -**Review found three more, all proven with repros and all fixed:** - -| Defect | Fix | -|---|---| -| The hub bound a joiner to seat `p2` *before* the room decided whether to seat them, and neither `Submit` nor `Resign` checked that the sender owned the seat. Anyone holding a room code — which is pasted into group chats by design — could resign or play on a seated player's behalf | Seating moved onto the room goroutine and happens only on success; `occupies` verifies the sending connection is the one in the seat | -| Cancelling the session context hard-closes the socket under `coder/websocket`, so the shutdown notice never left. Acceptance criterion (i) was failing | Read context separated from the teardown signal and rooted at `Background`, with a watcher that flushes before dropping the socket | -| A room whose game never started had no turn timer and no opponent, so nothing could ever end it; room creation was unlimited | A pre-game disconnect cancels the room; creation is rate-limited per session | - -Fixing those exposed a fourth, in `room.send`: a buffered channel and a cancelled context -are both ready, so `select` chose at random and half of all sends into a finished room -reported success. The caller then waited forever for a reply from a goroutine that had -stopped reading. The done-check is its own statement now. - -Also corrected: the join limiter keyed on the session id, which a client resets by -reconnecting — it keys on the client IP now, deliberately ignoring `X-Forwarded-For`, -which is attacker-controlled until phase 7 pins a trusted proxy. The static-file guard -compared string prefixes, so `/srv/webhooks` counted as inside `/srv/web`; it is a real -path-boundary check now. Room codes use rejection sampling instead of a modulo, which was -making the first eight letters 12% more likely. The handshake is one-shot. -`sanitizeNickname`'s seat parameter was dead — both players who sent an empty name became -"Người chơi 1" — so the collision is resolved in the room, where both names are known, -and now also covers two players who *choose* the same name. - -Every fix has a regression test, and the important ones were mutation-tested by -reintroducing the defect: the authorization guard fails 10/10 against the old code, the -shutdown guard 7/10 and the `room.send` guard 6/10 — both of those are timing races, so -detection is probabilistic by nature. - -**Deferred to phase 7:** the trusted-proxy policy for `X-Forwarded-For`, and a room -identity separate from the code if rematch is added, since codes are recycled on eviction. diff --git a/plans/260904-1125-noi-tu-web-game/phase-06-sveltekit-frontend.md b/plans/260904-1125-noi-tu-web-game/phase-06-sveltekit-frontend.md deleted file mode 100644 index cc3e539..0000000 --- a/plans/260904-1125-noi-tu-web-game/phase-06-sveltekit-frontend.md +++ /dev/null @@ -1,275 +0,0 @@ ---- -title: "Phase 6: SvelteKit Frontend" -status: done -phase: 6 -priority: P1 -effort: "5d" -dependencies: [4, 5] ---- - -# Phase 6: SvelteKit Frontend - -## Overview - -The playable UI: a SvelteKit SPA in JavaScript that connects over WebSocket, speaks -Protobuf, and renders the game. Vietnamese throughout, dark mode, local high score. This -phase delivers the complete vs-bot experience; phase 7 layers the online 1v1 flows on top. - -## Requirements - -**Functional** -- [x] Home screen: nickname input, play vs bot (Easy / Trung bình / Khó), or go to online play -- [x] Nickname persisted in `localStorage`, sent in `Hello`, and replaced by `accepted_nickname` from `Welcome` -- [x] Game screen: chain history, current syllable prompt, input, countdown ring, both scores -- [x] Rejections shown as specific Vietnamese messages, mapped from `RejectReason` -- [x] Game-over screen: result, final score, new-personal-best marker, rematch / home -- [x] Dark mode toggle, persisted; respects `prefers-color-scheme` on first visit -- [x] Personal best per difficulty in `localStorage` -- [x] Connection status indicator; automatic reconnect with backoff -- [x] In-app footer credit for the CC BY-SA 4.0 dictionary source, with links - -**Non-functional** -- [x] `adapter-static` build, served by the Go binary -- [ ] Mobile-first; usable one-handed on a phone with the keyboard up -- [x] No wordlist or dictionary data in the client bundle -- [ ] Vietnamese diacritic input (Telex/VNI IMEs) works — no input interception that breaks composition - -## Architecture - -**Stack:** SvelteKit 2 + Svelte 5 (runes: `$state`, `$derived`, `$effect`), plain JavaScript -with JSDoc types, `adapter-static`, Vite. Scaffold with `npx sv create`. - -``` -web/src/ - lib/ - proto/ # generated (phase 4) — do not hand-edit - ws/client.js # socket lifecycle, protobuf codec, reconnect backoff, ping/pong - ws/messages.js # thin senders: startBotGame(), submitWord(), createRoom(), joinRoom() - stores/game.svelte.js # $state game model, fed only by ServerMessage - stores/settings.svelte.js# nickname + theme + high scores, localStorage-backed - i18n/vi.js # ALL user-facing strings, incl. RejectReason → message map - components/ # ChainHistory, WordInput, CountdownRing, ScoreBoard, DifficultyPicker, - # ConnectionBadge, GameOverPanel, ThemeToggle, AttributionFooter - routes/ - +layout.svelte # theme application, footer - +page.svelte # home - play/+page.svelte # game screen (bot and PvP share it) - online/+page.svelte # room create/join (phase 7 fills this in) -``` - -**State rule:** the store is a projection of server messages. The client never decides -whether a word is valid, whose turn it is, or who won — it renders what the server sent. -The only client-owned state is theme, personal best, and the input box. - -**Countdown:** derived from `deadline_unix_ms` minus a clock offset measured by `Ping`/`Pong`, -re-evaluated on `requestAnimationFrame`. Purely cosmetic — the server decides expiry. - -**Reconnect:** exponential backoff (0.5s → 8s, jittered). On reconnect send `Hello` with the -stored `resume_token`; the server either restores the game or returns a typed error and the -UI falls back to the home screen with an explanation. - -**Vietnamese input:** bind on `change`/submit and read `event.target.value`; never -re-write the input's value mid-composition, and listen for `compositionstart`/`compositionend` -before any normalization — rewriting the field while a Telex IME is composing corrupts -diacritic entry. Normalization is the server's job anyway. - -**i18n:** every string lives in `lib/i18n/vi.js`, including the `RejectReason` map: - -| Reason | Message | -|---|---| -| `TOO_FEW_SYLLABLES` | "Từ phải có ít nhất 2 tiếng." | -| `WRONG_LINK` | "Từ phải bắt đầu bằng tiếng \"{syllable}\"." | -| `NOT_IN_DICTIONARY` | "Không tìm thấy từ này trong từ điển." | -| `ALREADY_USED` | "Từ này đã được dùng rồi." | -| `NOT_YOUR_TURN` | "Chưa đến lượt bạn." | -| `TIMEOUT` | "Hết giờ!" | - -**Theming:** CSS custom properties on `:root`, `[data-theme="dark"]` override, set from the -settings store before first paint to avoid a flash. - -## Related Code Files - -- Create: `web/` — SvelteKit scaffold (`package.json`, `svelte.config.js`, `vite.config.js`, `jsconfig.json`) -- Create: `web/src/lib/ws/client.js`, `web/src/lib/ws/messages.js` -- Create: `web/src/lib/stores/game.svelte.js`, `web/src/lib/stores/settings.svelte.js` -- Create: `web/src/lib/i18n/vi.js` -- Create: `web/src/lib/components/*.svelte` (per list above) -- Create: `web/src/routes/+layout.svelte`, `+page.svelte`, `play/+page.svelte`, `online/+page.svelte` -- Create: `web/src/app.css` — tokens, light/dark palettes -- Create: `web/src/lib/ws/client.test.js`, `web/src/lib/stores/game.test.js` (Vitest) -- Modify: `Makefile` — `make web`, `make web-dev` -- Modify: `server/internal/wsapi/server.go` — serve `NOITU_WEB_DIR` with SPA fallback - -## Implementation Steps - -1. Scaffold with `npx sv create web` (SvelteKit 2 / Svelte 5, JavaScript, Vitest, ESLint+Prettier); switch to `adapter-static` with `fallback: 'index.html'`. -2. Wire `make proto` output into `web/src/lib/proto`; add `@bufbuild/protobuf`. -3. `ws/client.js`: connect (`ws://` in dev via Vite proxy, same-origin `wss://` in prod), binary frames, decode to `ServerMessage`, dispatch to the store, ping/pong clock offset, backoff reconnect, `resume_token` in `sessionStorage`. -4. `stores/game.svelte.js`: `$state` model (phase, chain, currentSyllable, myTurn, deadline, scores, lastRejection, gameOver); one reducer per `ServerMessage` variant. -5. `stores/settings.svelte.js`: nickname, theme (system default, then explicit), `bestScore[difficulty]`, all `localStorage`-backed with try/catch so private-mode browsing still works. -6. Components: `ChainHistory` (scrolling word list, most recent pinned and highlighted, syllable-count badge on 3+ syllable words), `WordInput` (IME-safe, disabled when not your turn, autofocus on turn start), `CountdownRing` (SVG arc from the derived remaining time, colour shift under 5s), `NicknameInput` (20-char cap, mirrors the server rule), `ScoreBoard`, `DifficultyPicker`, `ConnectionBadge`, `GameOverPanel`, `ThemeToggle`, `AttributionFooter`. -7. Routes: home (nickname + mode + difficulty), `play` (the board, works for bot now and PvP in phase 7), `online` stub. -8. `i18n/vi.js` with every string and the rejection map; assert in a test that each proto `RejectReason` has an entry. -9. Personal best: on `GameOver`, compare and store per difficulty; show a "kỷ lục mới" marker. -10. `AttributionFooter`: names `minhqnd/dictionary`, links the repo and the CC BY-SA 4.0 deed — the user-visible half of the license obligation. -11. Vitest: store reducers for each server message, reconnect backoff schedule, rejection-map completeness, high-score persistence with `localStorage` throwing. -12. Verify the production bundle contains no word data (`grep` the built assets for known words) and record the bundle size. - -## Success Criteria - -- [ ] Full vs-bot game playable in a browser at all three difficulties — the protocol path is verified headlessly at all three; no browser playtest yet -- [ ] Words of 2, 3, and 4 syllables are all accepted and scored, with the syllable bonus visible -- [x] Nickname persists across reloads; the displayed name is always `accepted_nickname` from the server, never the raw input -- [x] Every rejection reason renders its specific Vietnamese message -- [ ] Countdown matches the server deadline within ~200ms; expiry is announced by the server, not the client -- [ ] Dark mode persists across reloads with no flash of the wrong theme -- [x] Personal best per difficulty persists and updates -- [ ] Killing the server mid-game shows a connection state and reconnects when it returns -- [ ] Diacritics typed with a Telex IME enter correctly on desktop and mobile -- [x] `npm run build` → `adapter-static` output served correctly by the Go binary, deep links included -- [x] Built assets contain no dictionary words -- [x] Attribution footer present with working links -- [x] `npm test` green - -## Risk Assessment - -| Risk | Signal | Response | -|---|---|---| -| IME composition broken by input handling | Diacritics mangled while typing | Never rewrite the input value during composition; guard with `compositionstart`/`end`. Explicitly tested on a real mobile keyboard, not just a desktop emulator | -| Countdown drift makes a legitimate move look late | Player sees 2s left, server says timeout | Clock offset from `Ping`/`Pong`; the ring visually settles ~300ms early so the client never claims more time than the server allows | -| Svelte 5 runes in `.svelte.js` store files misused | Reactivity silently stops updating | Stores use the documented `.svelte.js` rune-module pattern; store reducers are unit-tested outside components | -| Protobuf runtime bloats the bundle | Bundle budget exceeded | Measured in step 12; `protobuf-es` is tree-shakeable and only generated messages are imported | -| `localStorage` unavailable (private mode / blocked) | Crash on load | Every read/write wrapped in try/catch with a working in-memory default | -| Dev/prod WS URL divergence | Works in `npm run dev`, breaks when served by Go | Same-origin `wss://` resolved from `location` in prod; Vite proxy only in dev; both paths exercised before phase 7 | - -## Phase 6 Outcome (2026-09-05) - -The vs-bot UI is built, typed, tested, and served by the Go binary. 117 JavaScript tests pass -and `svelte-check` reports no errors across 368 files. - -### What was verified, and how - -| Claim | Evidence | -|---|---| -| The binary serves the SPA, deep links included | `curl` against a running server: `/`, `/play?difficulty=2` and `/online` all return `index.html`; hashed assets and the favicon return their own content types | -| Path traversal cannot escape the bundle | `/../server/go.mod` is normalised to a redirect; the percent-encoded form is refused with 400 | -| A bot game plays end to end at every difficulty | A headless client built from the same generated types played three moves against the bot at Easy, Medium and Hard against a fixture dictionary, with alternating turns and correct running scores | -| The nickname shown is the server's | The handshake returned `accepted_nickname` and the store adopts it | -| No wordlist reaches the browser | A bundle test greps every built file for dictionary words and asserts a size budget | - -The bundle is 188,227 bytes across 28 files. - -### Deliberate design points - -**The store is a reducer over `ServerMessage` and nothing else.** Validity, turn order and the -result are read from the wire, never computed. That is what lets one screen serve both the bot -and, in phase 7, online play. - -**The word field is uncontrolled.** A Telex or VNI input method composes a diacritic across -several keystrokes, and writing the value back on each keystroke cancels the composition. The -field is read on submit and cleared only there, where composition has already ended. - -**The countdown settles 300ms early.** A ring showing time left while the server has already -timed the player out reads as the game cheating; reaching zero a fraction early reads as -rounding. - -**`Hello` is sent before the open status is announced.** The server refuses everything before -the handshake, so a listener reacting to "connected" by sending a message would otherwise race -it. Ordering it explicitly turns a timing accident into a contract. - -### Defects found by review - -Review found one blocking defect and four user-visible breakages, all in code no test was -driving. The pattern behind the worst of them is worth naming: **the game lifecycle was -inferred from the game model instead of being owned.** - -**Every rematch started two bot games, and the second killed the first.** The screen sent -`StartBotGame` from an effect whenever the board was idle. Clearing the board for a rematch is -exactly that condition, so the rematch button sent one request and the inference sent another. -The server allocates a room per request and does not deduplicate, so the player got two rooms -sharing one socket: the abandoned room kept its turn timer, overwrote the turn sequence that -the live game's submissions were tagged with, and about one turn later ended the game in a loss -nobody could explain. A request is now stored intent — `web/src/lib/stores/bot-session.svelte.js` -— and clearing the board cannot mean anything. Twelve tests cover it, because the absence of a -test over this wiring is why it shipped. - -**Arriving at the board with a finished game on screen never started a new one.** Same root -cause: nothing reset the store on entry, so the header link or the browser Back button left a -stale phase behind and the idle condition never held. The screen now owns the socket and the -game for as long as it is mounted, and gives the current game up on the way out rather than -leaving a room whose timer keeps running somewhere the player cannot see. - -**A word typed during a reconnect vanished silently.** The input stayed enabled while the socket -was down, the send returned false, and the field cleared anyway. The field is now gated on the -connection as well as the turn, and only clears a word that actually went out. - -**A protocol bump would have put every open tab into a reconnect loop.** The backoff reset when -the socket opened, but the server accepts the connection and *then* refuses an unspeakable -`Hello`, so every retry looked like a success and restarted from the shortest delay. The -backoff now resets on `Welcome`, and `protocol_version_mismatch` stops reconnection outright. - -**The opening turn counted down against the raw device clock.** The first clock probe was -scheduled a full interval out, but `GameStarted` arrives one round trip after `Hello`. A device -clock behind the server would have shown more time than the server allowed — the exact thing -the settle margin exists to prevent. The probe now goes out with the handshake. - -Also fixed from the review: cache headers on the static bundle, where an `index.html` cached -across a deploy names hashed assets that no longer exist and the app loads into a blank page; -a record-keeping effect that read and wrote the same reactive state; an animation loop that ran -at 60fps between turns; and the difficulty picker, which claimed a radio role without radio -keyboard behaviour and now uses real radio inputs. - -### Defects found while building - -**The bundle guard was vacuous on its first negative test.** Planting an unused dictionary word -in the string table did not fail it, because Rollup dropped the unused property. Re-testing with -a rendered string made it fail correctly. It now also decodes `\uXXXX` escapes, so a minifier -configured to emit ASCII cannot hide the words from a substring search. - -**`jsdom` broke the phase-4 fixture test.** Making it the global test environment rewrote -`import.meta.url` to an `http:` URL, so the cross-language fixture reader could not resolve its -path. The DOM is now opted into per file by the two suites that need it. - -**`svelte-check` found a required prop that no caller passed.** `DifficultyPicker` declared -`onselect` as required while the home screen only binds `value`. The prop is optional now. - -### Guards added after review - -| Guard | What it would catch | -|---|---| -| `tests/bot-session.test.js` | A rematch sending two requests, or a result scored twice | -| `tests/error-codes.test.js` | A server error code with no Vietnamese message, read from the Go source that emits them; it also fails on copy for a code the server cannot send | -| `TestStaticCacheHeaders` | The shell becoming cacheable across a deploy | -| Build freshness in `tests/bundle.test.js` | The wordlist check reading a stale bundle | -| Handshake ordering in `tests/ws-client.test.js` | `Hello` losing its place ahead of the open status | - -The error-code guard and the wordlist guard were both negative-tested by breaking what they -protect and watching them fail for the right reason. The bundle budget was 1.5 MB against a -188 KB bundle, which caught nothing; it is 400 KB now. - -### Deviations from the plan - -- Scaffolded by writing the config files directly rather than `npx sv create`, which is - interactive. ESLint and Prettier were not added; `svelte-check` is wired into `make test-web` - instead. -- Tests live in `web/tests/`, following the layout phase 4 established, not beside their - sources. -- `npm test` now runs `vite build` first. The bundle check reads built output, and a stale - build would let it pass over code that no longer exists. This also means the proto workflow's - JavaScript step builds the frontend, which needs no dictionary and stays fast. - -### Not verified - -Five criteria need a real browser and are left unchecked: one-handed mobile use, Telex -diacritic entry, the absence of a theme flash, reconnect after the server is killed mid-game, -and the rematch button. The logic behind the last three is unit-tested — the pre-paint theme -script, the jittered backoff schedule, and the game-request latch — but none has been watched -happen. The rematch path is worth naming separately: it is where review found the blocking -defect, and the fix is verified by unit tests over the latch rather than by playing a game. - -The countdown criterion is no longer known to be violated on the opening turn, since the clock -probe now goes out with the handshake, but the ~200ms figure has not been measured. - -A resumed session shows the opening word and the last move rather than the whole chain. That -is the server's replay contract, which rebuilds the position from the engine rather than -replaying a recorded stream; the client renders what it is sent. diff --git a/plans/260904-1125-noi-tu-web-game/phase-07-online-1v1-and-release.md b/plans/260904-1125-noi-tu-web-game/phase-07-online-1v1-and-release.md deleted file mode 100644 index 0ec887a..0000000 --- a/plans/260904-1125-noi-tu-web-game/phase-07-online-1v1-and-release.md +++ /dev/null @@ -1,277 +0,0 @@ ---- -title: "Phase 7: Online 1v1 and Release" -status: done -phase: 7 -priority: P1 -effort: "4d" -dependencies: [5, 6] ---- - -# Phase 7: Online 1v1 and Release - -## Overview - -Complete the online 1v1 experience end to end in the browser — room codes, waiting room, -opponent presence, reconnect UX, rematch — then verify the whole game with browser E2E tests -and ship it: CI, container image, deployment, and documentation. - -## Requirements - -**Functional** -- [x] Create a room → shareable 6-character code and a copyable invite link -- [x] Join by code, or by opening an invite link with the code prefilled -- [x] Waiting room until the opponent arrives; leaving cleans the room up -- [x] Both players' nicknames shown in the waiting room and scoreboard (server-sanitized values only) -- [x] Live opponent state: their turn, their timer, their disconnect and reconnect -- [x] Resign, and rematch in the same room after a game ends -- [x] Clear Vietnamese error states: room not found, room full — a room whose game has started is full, so there is no third state - -**Non-functional** -- [x] E2E coverage of both modes with two real browser contexts -- [ ] CI: Go tests + race, JS tests, `buf lint`/`breaking`, generated-code drift check, builds -- [x] Deployable artifact: container image with the binary, the static frontend, and `noitu.db` -- [x] README documents setup, the license split, and deployment - -## Architecture - -No new server concepts — phase 5 already implements rooms, codes, reconnect, and resign. -This phase is the frontend surface for them plus release engineering. - -**Invite link:** `https:///online?code=ABC123`. The `online` route reads the query -param, prefills, and auto-joins when the code is well-formed. Room codes use an unambiguous -alphabet (no `0/O/1/I`) and are displayed grouped for reading aloud. - -**Online screen states** - -``` -idle ──create──► waiting (code shown, copy button) - │ └──opponent joins──► playing - └──join(code)──► playing | error(room_not_found | room_full | already_started) - -playing ──opponent disconnects──► banner "Đối thủ mất kết nối… (Ns)" - ├─ they return ──► resume playing - └─ grace expires ──► GameOver(END_OPPONENT_LEFT) -``` - -**Rematch:** after `GameOver` in a PvP room, either player may request a rematch; the room -resets its engine with a fresh opening word when both accept, or closes on decline/timeout. -Requires a small additive protocol change — `RequestRematch` / `RematchState` — which is why -it lands here rather than in phase 4: additive tags only, per the phase-4 compatibility rule. - -**Deployment — Docker image on a VPS behind a reverse proxy (validated decision):** - -``` -Dockerfile (multi-stage) - node → npm ci && npm run build → /web/build - go → CGO_ENABLED=0 go build ./cmd/... → /noitu-server - data → curl the 179 MB dictionary.db → build-dictionary → /data/noitu.db - final → distroless/static + binary + /web + /data/noitu.db + /data/LICENSE + ATTRIBUTION -``` - -`CGO_ENABLED=0` works because the SQLite driver is `modernc.org/sqlite` (pure Go) — the -payoff for that phase-1 decision. The 179 MB upstream download happens **only in the data -builder stage**, so it never reaches the final image; `noitu.db` is copied in as its own -layer, keeping the CC BY-SA 4.0 artifact physically distinct from the Apache-2.0 binary. - -**CI data strategy (consequence of not republishing a derived DB):** CI must never download -179 MB per run. - -| Job | Data used | -|---|---| -| Unit tests (every push/PR) | Small fixture DB built in-test from a checked-in word sample (phase 2 helper) | -| E2E tests (every push/PR) | Same fixture DB, ~200 curated words — enough to script deterministic games | -| Full dictionary build (manual / nightly) | Real upstream, with the download cached by release tag | -| Docker image build (release only) | Real upstream, inside the builder stage | - -## Related Code Files - -- Modify: `proto/noitu/v1/game.proto` — add `RequestRematch`, `RematchState` (new tags only) -- Modify: `server/gen/...`, `web/src/lib/proto/...` — regenerate -- Modify: `server/internal/wsapi/room.go` — rematch handling, room reset -- Modify: `web/src/routes/online/+page.svelte` — replace the phase-6 stub with the full create/join/waiting/error flow -- Create: `web/src/lib/components/RoomCodePanel.svelte`, `WaitingRoom.svelte`, `OpponentStatus.svelte`, `RematchPrompt.svelte` -- Modify: `web/src/lib/stores/game.svelte.js` — room/opponent/rematch state -- Modify: `web/src/lib/i18n/vi.js` — online-mode strings -- Create: `e2e/bot-game.spec.js`, `e2e/pvp-game.spec.js`, `e2e/reconnect.spec.js` (Playwright) -- Create: `playwright.config.js` -- Create: `Dockerfile`, `.dockerignore` -- Create: `.github/workflows/ci.yml` -- Modify: `README.md` — quickstart, architecture, deployment, license split -- Create: `docs/deployment.md` - -## Implementation Steps - -1. Add `RequestRematch` / `RematchState` to the proto with new tags; regenerate both targets; extend the round-trip tests. -2. Server: rematch in `room.go` — both-accept resets the engine with a new opening word and re-emits `GameStarted`; decline or timeout closes the room. Cover with a room test. -3. `online/+page.svelte`: the state machine above, code input (uppercase, alphabet-restricted, paste-friendly), `?code=` prefill and auto-join. -4. `RoomCodePanel`: large grouped code, copy button, copy invite link, Web Share on mobile where available. -5. `WaitingRoom`: opponent-pending state with a cancel that leaves the room cleanly. -6. `OpponentStatus`: turn indicator, disconnect banner with the grace countdown, reconnect confirmation. -7. `RematchPrompt` on the game-over panel for PvP; bot mode keeps a plain "chơi lại". -8. Vietnamese strings for every new state, including the three join errors. -9. Playwright setup: build the frontend, run the real server against the **~200-word fixture DB** (never the real dictionary — CI must not download 179 MB), run specs against it. -10. E2E `bot-game.spec.js`: pick Hard, play scripted valid and invalid words, assert rejection messages, play to game over, assert the score and personal-best marker. -11. E2E `pvp-game.spec.js`: two browser contexts with different nicknames, create + join by code, assert each side shows the other's sanitized nickname, alternate turns, assert each side sees the other's word, resign ends it correctly. -12. E2E `reconnect.spec.js`: drop one context's socket mid-game, assert the opponent's disconnect banner, restore within grace, assert the board matches on both sides. -13. `Dockerfile` multi-stage as above; verify the image runs with only `NOITU_DB_PATH` set. -14. `ci.yml`: `go vet`, `go test ./... -race`, `npm test`, `npm run build`, `buf lint`, `buf breaking`, generated-code drift check, Playwright — all on the fixture DB. Separate release-only job builds the Docker image. Add an assertion that the built image contains `data/LICENSE` and `data/ATTRIBUTION.md`. -15. `README.md`: what the game is, quickstart (**`make fetch-dict` downloads 179 MB once**, then `make dict`), `make` targets, architecture diagram, **License** section (Apache-2.0 code / CC BY-SA 4.0 data with attribution), and a link to `data/ATTRIBUTION.md`. -16. `docs/deployment.md`: env vars (incl. `NOITU_TURN_LIMIT=20s`, `NOITU_ALLOWED_ORIGINS`), TLS/`wss://` behind a reverse proxy (`Upgrade`/`Connection` headers, proxy read timeout longer than the WS keepalive, buffering disabled), health check, and how `noitu.db` reaches the container. - -## Success Criteria - -- [x] Two independent browser contexts play a full game via a shared room code; two physical machines untested -- [x] Invite link opens straight into the room -- [x] Both join errors the protocol has show correct Vietnamese messages, plus a malformed code caught before it is sent -- [x] Disconnect shows the opponent a grace countdown; return inside it resumes with identical boards; expiry awards the win -- [x] Rematch restarts in the same room with a new opening word -- [x] Resign ends the game immediately with the right winner -- [x] Playwright suite green: bot game, PvP game, reconnect -- [ ] CI green on all steps including the generated-code drift check, without ever downloading the 179 MB upstream DB -- [x] `docker run` serves a playable game; the image contains `data/LICENSE`, `data/ATTRIBUTION.md` and `NOTICE` but no upstream file — verified against a fixture-dictionary build, not an upstream one -- [x] README license section and in-app attribution agree with `data/ATTRIBUTION.md` -- [ ] Full pass over `plan.md` success criteria — every box checkable - -## Risk Assessment - -| Risk | Signal | Response | -|---|---|---| -| Reverse proxy breaks WS upgrade in production | Works locally, 400/502 on deploy | `docs/deployment.md` states the required proxy headers and timeouts; the health check plus a post-deploy WS smoke test catch it before users do | -| Playwright PvP tests flake on timing | Intermittent CI red | Drive by deterministic server events (wait for the turn indicator), never fixed sleeps; run the suite with a long turn limit via env | -| Rematch protocol addition breaks phase-6 clients | Decode errors in an already-running client | Additive tags only; `buf breaking` in CI is the guard, and `protocol_version` lets the server reject a stale client cleanly | -| Room codes guessable enough to join a stranger's game | Reports of uninvited joins | `crypto/rand` codes plus per-IP join rate limiting from phase 5; 32^6 space with rate limiting makes scanning impractical | -| CC BY-SA obligations lost during packaging | Image ships without `data/LICENSE` | The Docker copy step includes the whole `data/` directory, and a CI assertion checks `data/LICENSE` and `data/ATTRIBUTION.md` exist in the built image | -| Scope creep into accounts/leaderboards at the finish line | New requests during release work | Explicit non-goals in `plan.md`; log them as post-v1 items instead | - -## Phase 7 Outcome (2026-09-05) - -Online 1v1 is playable end to end, the whole game is covered by a browser suite, and the -result ships as one 24 MB container image. - -### What was verified, and how - -| Claim | Evidence | -|---|---| -| Two players join by code and alternate turns | Two browser contexts against the real binary; each side sees the other's sanitized nickname and the other's words | -| An invite link opens straight into the room | The second context only opens a URL — no code typed, no button pressed | -| A rematch restarts the same room | One acceptance shows the other player a prompt; the second starts a new game with a fresh opening word and the room code unchanged | -| A disconnect is announced and survivable | Taking a player's page away raises the opponent's banner; returning inside the window restores the same position and clears it | -| An opponent who never returns forfeits | The grace window expires and the win is awarded, with the reason shown | -| The image runs and carries its obligations | `docker run` serves the app and a deep link; the image contains `data/LICENSE`, `data/ATTRIBUTION.md` and `NOTICE`, and no upstream database | -| The difficulty ladder is real | The simulation suite from phase 3 reports Hard beating Easy 94 games to 6, Medium beating Easy 93 to 7, and Hard beating Medium 78 to 22 | - -21 browser tests, 139 JavaScript unit tests, and the Go suite under `-race` all pass. - -### Deliberate design points - -**The room now outlives its game, but only when there is someone to ask.** A finished game -opens a rematch offer when both seats still hold live connections. A bot room closes exactly -as before, because there is nothing to negotiate with a bot and keeping it alive would leak a -goroutine and an engine per finished game. - -**There is no decline message.** Leaving is the decline, and the server already learns about -that from the socket closing. One message and one timeout cover every way a rematch does not -happen, which is a smaller protocol and one less state to get wrong. - -**`turn_seq` no longer restarts at one.** A rematch reuses the same connections, so a -submission still in flight from the previous game could otherwise match a turn in the new one -and be applied to it. - -**The fixture dictionary goes through the real builder.** `build-dictionary --words` reads a -checked-in list and runs it through the same filter, alias and write path production uses, so -a test database cannot drift into being shaped differently from what the server loads. Its -graph has exactly one syllable productive enough to open on, which is what makes a scripted -game deterministic without pinning the bot's replies. - -### Defects found by building the browser suite - -**A player who refreshed mid-game could not get back in.** The online screen waited for the -player to ask for a room before connecting — correct for someone who has just arrived and is -still typing their name, wrong for a tab that already holds a session. It now reconnects -immediately when a resume token is present. - -**The opponent's disconnect banner never cleared.** The server restored the seat but told -nobody, so the other player watched "đối thủ mất kết nối" for someone already playing again. -A resume now announces itself to the opponent. - -**A dead socket was invisible to the client.** Nothing noticed a connection that failed -without closing, so the page kept showing a live connection and a running countdown over a -socket nothing could reach. The client now treats silence longer than three ping intervals as -a dead socket and reconnects. - -**The nickname typed on the online screen never reached the server.** The socket opened on -arrival and `Hello` carries the name once, so both players were introduced under whatever was -stored before they got there. Connecting when the player actually asks for a room fixed it. - -### Two tools that did not work, and why - -**Offline emulation does not cut a loopback socket.** A test that "went offline" kept playing -happily against the local server. Routing the WebSocket cuts it for real. - -**A routed socket's close does not reliably reach the server.** The page observes it, but the -grace window never starts. That makes socket routing the right tool for what the client does -about a dead connection and the wrong one for what the server does about a missing player; the -latter tests take the page away instead. Both limitations are recorded in `e2e/socket-cut.js` -so the next person does not rediscover them. - -### Deviations from the plan - -- Playwright lives under `web/` rather than the repository root, so there is still one npm - package rather than two. -- There are two join errors, not three: a room whose game has started is full, and the server - has no separate code for it. A malformed code is caught in the client before it is sent. -- CI is split across two workflows. `proto.yml` keeps the wire contract, `ci.yml` owns the - tests and the image, and the duplicated test steps were removed from the former. -- The image is built on every push against the fixture word list, not only on release. An - image built only at release time is an image that breaks at release time, and the build - argument that makes this cheap already existed for local testing. - -### Defects found by review - -Review reproduced three defects, two of them consequences of the new state this phase -introduced: a room that outlives its game. - -**A game that ended on the turn clock never offered a rematch, and the button was still -there.** The offer was opened from the message arm of the room loop, so the most common -natural ending — running out of time — closed the room with nothing to accept. The player -pressed "chơi lại" and got an error. The end-of-game decision now sits after the whole select, -so every way a game can end reaches it, and the online game-over panel no longer shows a -rematch button at all: the prompt appears only while an offer is actually open, so it is the -only thing that can ask. - -**A second connection presenting the same resume token disconnected the player who was still -there.** `resumeFrom` retired the old connection as soon as the room had *accepted the -message*, but the room decides asynchronously and refuses a resume into a finished game — which -is exactly the state during a rematch offer. A duplicated tab therefore ended the room. The -room now retires the old connection only once it has agreed to the swap. Before this phase the -window did not exist, so a refused resume was unreachable. - -**Rooms leaked when one connection created several.** `attach` overwrote the session's room -pointer and nothing told the old room, which then parked in `select` forever holding a -goroutine and a room code. Rate limiting bounds the rate, not the total, and connections are -free, so this was unbounded on a public endpoint. `attach` now releases the room it is leaving. -This one predates the phase; it is fixed here because this is the phase that puts the server on -the internet. - -Also fixed: `RequestRematch` had no rate limit despite being the only client message that fans -out to both players, so a burst could fill the opponent's outbox until the server closed their -session. A stale disconnect notice was being treated as a rematch decline, which needed -`handleDisconnect` to stop conflating "this notice applied" with "a game is still running". -The client's liveness check could fire on a throttled background tab and close a healthy -socket, so it now ignores a tick that was itself late. And a stale resume token opened the -lobby with an error the player did nothing to cause. - -Each of the three defects has a regression test, and the two that were reachable through the -room loop were negative-tested by reverting the fix and watching them fail. - -### Not verified - -CI has not run on GitHub. Every step passes locally, including the image build and its licence -assertions, but the workflows themselves are unexercised. - -The image was built from the fixture word list, not the 179 MB upstream release. The -downloading branch of the `Dockerfile` is the same shape, pinned by the same checksum the -`Makefile` uses, but it has not been run here. - -Two players on two physical machines have not played. Two independent browser contexts have, -which exercises everything except the network between them. diff --git a/plans/260904-1125-noi-tu-web-game/plan.md b/plans/260904-1125-noi-tu-web-game/plan.md deleted file mode 100644 index 7007d0a..0000000 --- a/plans/260904-1125-noi-tu-web-game/plan.md +++ /dev/null @@ -1,381 +0,0 @@ ---- -title: "Noi Tu Web Game" -description: "Vietnamese nối từ web game — SvelteKit frontend, Go backend, WebSocket + Protobuf, server-authoritative dictionary over SQLite. Vs-bot and online 1v1." -status: complete -priority: P1 -effort: "~3-4w" -tags: [game, sveltekit, go, websocket, protobuf, sqlite, vietnamese] -created: 2026-09-04 -blockedBy: [] -blocks: [] ---- - -# Noi Tu Web Game - -## Overview - -Web implementation of **nối từ**, the Vietnamese word-chain game: a player submits a -meaningful word of **at least 2 syllables** whose **first syllable equals the previous -word's last syllable**. No reuse, timed turns. Loss on timeout, invalid word, wrong link, -repeat, or no legal move remaining. - -Two modes ship in v1: **vs bot** (3 difficulties) and **online 1v1**. Both run through the -same server-authoritative engine — the browser never holds the wordlist, so validation -cannot be bypassed and the bot and PvP paths share one code path. - -**Stack (user-selected):** SvelteKit (JavaScript) frontend · Go backend · WebSocket -transport with Protobuf framing · dictionary served from SQLite server-side. - -**Research basis:** [`plans/reports/research-260904-1058-noi-tu-game.md`](../reports/research-260904-1058-noi-tu-game.md) - -## Goals - -| # | Goal | Priority | -|---|------|----------| -| 1 | Correct, server-authoritative nối từ rules incl. Vietnamese text normalization | P1 | -| 2 | Reproducible dictionary pipeline from `minhqnd/dictionary` with correct CC BY-SA 4.0 compliance | P1 | -| 3 | Typed WS protocol (Protobuf) shared by Go and JS, single source of truth | P1 | -| 4 | Playable vs-bot mode with 3 difficulty levels | P1 | -| 5 | Online 1v1 with nicknames, room codes, turn timer, reconnect | P1 | -| 6 | Vietnamese UI, dark mode, local high score | P2 | -| 7 | Single deployable Go binary, shipped as a Docker image | P2 | - -## Non-goals (v1) - -- Word definitions / meanings display (explicitly out of scope per user selection) -- Accounts, auth, persistent profiles, server-side leaderboards -- 4-player *đấu trường* mode -- Native mobile apps -- Player-submitted word additions / dictionary moderation UI -- Redistributing our derived dictionary as a downloadable artifact (see Data Distribution) - -## Architecture - -``` -┌──────────────────────────────┐ ┌────────────────────────────────────┐ -│ SvelteKit SPA (adapter- │ WSS │ Go server (single binary) │ -│ static, Svelte 5 runes) │ ◄─────► │ │ -│ │ Protobuf│ wsapi ── hub ── room(s) │ -│ lib/ws connection+codec │ binary │ │ │ -│ lib/proto protobuf-es gen │ frames │ ├── game.Engine │ -│ lib/stores game state │ │ └── bot.Bot │ -│ routes/ UI (vi, dark mode)│ │ │ │ -└──────────────────────────────┘ │ dictionary.Store ──┘ │ - localStorage: nickname, high score, │ │ read-only SQL │ - theme │ data/noitu.db (CC BY-SA 4.0) │ - └────────────────────────────────────┘ - ▲ built locally by - server/cmd/build-dictionary (Go) - ▲ reads - dictionary.db (179 MB, downloaded from - minhqnd/dictionary release v2.0.0) -``` - -**Key decisions** - -| Decision | Choice | Rationale | -|---|---|---| -| Authority | Server validates every move; client is a view | User chose server-side dictionary; also the only way PvP is cheat-resistant | -| Word length | **≥ 2 syllables**, link on first↔last syllable | User decision. Closer to casual play; keeps 3+ syllable compounds in the corpus | -| Bot location | Server-side, same engine as PvP | One rule implementation, no duplication (DRY); bot "thinking delay" is server-timed | -| Turn limit | **20s**, identical for bot and PvP | One constant, one code path; `NOITU_TURN_LIMIT` overrides | -| Player identity | **User-typed nickname**, no accounts | User decision. Stored in `localStorage`, sanitized server-side | -| WS library (Go) | `github.com/coder/websocket` v1.8.15 (ISC) | `gorilla/websocket` archived 2022 and panics on concurrent writes; coder/websocket is context-aware and handles concurrent writes | -| Protobuf (JS) | `@bufbuild/protobuf` v2.14.1 + `protoc-gen-es` | Only fully conformant JS impl, ESM/tree-shakeable, small browser bundle | -| Protobuf (Go) | `protoc-gen-go` via same `buf.gen.yaml` | One schema, two targets, generated in CI | -| Go module path | `github.com/tiennm99dev/noitu/server` | Matches the actual git remote | -| SQLite driver | `modernc.org/sqlite` (CGo-free) | Keeps `CGO_ENABLED=0` cross-compilation and a distroless image; read-only lookups are not CGo-bound | -| Dictionary DB | Loaded at runtime from `data/noitu.db`, **not** `go:embed` | Keeps CC BY-SA 4.0 data a separate artifact from Apache-2.0 code — cleanest license boundary | -| Tone variants | Alias table built offline (`hoà`→`hòa`) | Precomputed lookup beats a fragile runtime tone-placement algorithm (KISS) | -| Room state | In-memory, no DB | v1 has no persistence requirement; restart drops live games (accepted, documented) | -| Deployment | Docker image on a VPS behind a reverse proxy | Most portable; phase 7 ships a distroless image and proxy docs | - -## Data Distribution - -**Decision (validated):** we do **not** republish a derived dictionary. Every build downloads -`dictionary.db` from the upstream release and derives `data/noitu.db` locally. - -Upstream asset: [`minhqnd/dictionary` release v2.0.0 → `dictionary.db`](https://github.com/minhqnd/dictionary/releases/download/v2.0.0/dictionary.db) — **179 MB**. - -Consequences, all handled explicitly: - -| Consequence | Handling | -|---|---| -| 179 MB is too large to fetch on every CI run | CI runs tests against a **small fixture DB** built from a checked-in word sample (phase 2). Only a manually-triggered/nightly job downloads the real upstream and builds the full DB | -| Docker build needs the upstream DB | Downloaded in the **builder stage** only; the final image carries just the derived `noitu.db`, so the 179 MB never ships | -| `data/noitu.db` is a local build artifact | Git-ignored. `make dict` is a documented prerequisite for running the real server | -| The derived DB *is* distributed inside our Docker image | CC BY-SA 4.0 still applies to that image layer — attribution files ship with it, asserted in CI (phase 7) | - -## Licensing (mandatory — CC BY-SA 4.0 share-alike) - -`minhqnd/dictionary` is **dual-licensed**: MIT for application code, **CC BY-SA 4.0 for the -dictionary data**. We consume only the data, so share-alike applies to our derived wordlist -wherever we distribute it (notably the Docker image). - -| Artifact | License | File | -|---|---|---| -| All source code in this repo | Apache-2.0 (existing) | `LICENSE` | -| Derived dictionary `data/noitu.db` and the build script's output | **CC BY-SA 4.0** | `data/LICENSE` | -| Attribution + list of modifications | — | `data/ATTRIBUTION.md` | -| Root pointer to the split | — | `NOTICE`, README section | -| User-visible credit | — | in-app footer with link | - -Attribution must name: `minhqnd/dictionary`, its own upstream sources (Wiktionary, `vntk/dictionary`), -the CC BY-SA 4.0 license URL, and **what we changed** (filtered to Vietnamese entries of ≥2 -syllables, NFC-normalized, tone-variant aliases added, first/last syllable columns and indexes -added, all non-`vi` languages and all definitions/translations dropped). - -## Phases - -| # | Phase | Status | Depends on | -|---|-------|--------|-----------| -| 1 | [Foundations and Data Pipeline](./phase-01-foundations-and-data-pipeline.md) | Complete | — | -| 2 | [Go Dictionary and Normalization](./phase-02-go-dictionary-and-normalization.md) | Complete | 1 | -| 3 | [Go Game Engine and Bot AI](./phase-03-go-game-engine-and-bot-ai.md) | Complete | 2 | -| 4 | [Protobuf Contract and Codegen](./phase-04-protobuf-contract-and-codegen.md) | Complete | 1 | -| 5 | [Go WebSocket Server and Rooms](./phase-05-go-websocket-server-and-rooms.md) | Complete | 3, 4 | -| 6 | [SvelteKit Frontend](./phase-06-sveltekit-frontend.md) | Complete | 4, 5 | -| 7 | [Online 1v1 and Release](./phase-07-online-1v1-and-release.md) | Complete | 5, 6 | - -Phases 2-3 and 4 are independent after phase 1 and can run in parallel if desired. - -## Target Repository Layout - -``` -LICENSE # Apache-2.0 — code only -NOTICE # points at data/ licensing -README.md -buf.yaml buf.gen.yaml -proto/noitu/v1/game.proto # single protocol source of truth -data/ - LICENSE # CC BY-SA 4.0 full text - ATTRIBUTION.md - dictionary.db # upstream, 179 MB, git-ignored, downloaded - noitu.db # derived build artifact, git-ignored -server/cmd/build-dictionary/ # Go: dictionary.db -> noitu.db -server/ # module github.com/tiennm99dev/noitu/server - go.mod - cmd/noitu-server/main.go - internal/vietnamese/ # normalize.go, syllable.go - internal/dictionary/ # store.go — read-only SQLite lookups - internal/game/ # engine.go, state.go, rules.go - internal/bot/ # bot.go, strategy_*.go - internal/wsapi/ # hub.go, room.go, session.go, codec.go - gen/noitu/v1/ # protoc-gen-go output -web/ - src/lib/proto/ # protobuf-es output - src/lib/ws/ # socket client, reconnect, codec - src/lib/stores/ # game state (Svelte 5 runes) - src/lib/components/ - src/routes/ -``` - -## Success Criteria - -- [x] `server/cmd/build-dictionary` reproducibly turns the upstream `dictionary.db` into `data/noitu.db`; entry count and ≥2-syllable purity asserted -- [x] `data/LICENSE`, `data/ATTRIBUTION.md`, `NOTICE`, README license section, and in-app credit all present and consistent -- [x] Go engine unit tests cover: wrong link, unknown word, reuse, single-syllable input, timeout, no-legal-move, and the tone-variant cases `hoà/hòa`, `thuý/thúy`, `quí/quý` -- [x] Words of 2, 3, and 4 syllables all accepted and chain correctly on first↔last syllable -- [x] One `.proto` generates working Go and JS clients; no hand-written message types -- [x] Vs-bot playable end to end at all 3 difficulties; Hard bot wins measurably more than Easy over 100 simulated games -- [x] Online 1v1: two browsers join by room code with chosen nicknames, alternate turns, server-enforced 20s timer, correct win/loss, reconnect within grace window restores the game -- [x] Vietnamese UI throughout; nickname and high score persist, checked in a real browser. The dark-mode toggle persists by the same mechanism but has only been unit-tested -- [x] `go build` produces one binary; `web` builds to static assets served by that binary -- [x] No wordlist reachable from the client bundle (verified by inspecting the built assets) -- [ ] CI runs green without ever downloading the 179 MB upstream DB — the workflows are written and every step passes locally, but they have not run on GitHub - -## Risk Assessment - -| Risk | Signal it happened | Response | -|---|---|---| -| Upstream `dictionary.db` yields too few clean ≥2-syllable entries (<40k) | Build script count assertion fails | Union with Viet74K multi-syllable filter (see research report §5) — decided before implementation, not mid-phase | -| Tone-variant aliasing misses real-world spellings | Players report valid words rejected | Alias table is data, not code: regenerate with an expanded rule set and rebuild `noitu.db`; add a rejected-word log to find gaps | -| CC BY-SA share-alike misapplied to code | License review flags the Apache/CC mix | Data stays a runtime-loaded separate artifact, never `go:embed`-linked; boundary documented in `NOTICE` and asserted in the image | -| Hard bot is unbeatable, players quit | Playtest win rate ≈0% vs Hard | Cap Hard's killer-syllable play rate; the difficulty ladder is tunable constants, not structure | -| In-memory rooms lost on deploy/restart | Live games drop | Accepted for v1 and stated in-app ("máy chủ đang khởi động lại"); persistence is a post-v1 item | -| Protobuf schema churn breaks a deployed client | Client decode errors after deploy | Additive-only field changes, reserved tags, version field in the handshake; server rejects unknown protocol versions with a clear message | -| Nicknames become an abuse surface (shown to strangers) | Offensive names reported | Server-side sanitization: length cap, control-char strip, whitespace collapse, empty → auto-name. A denylist is a cheap post-v1 addition if needed | -| Allowing 3+ syllable words admits phrases that feel wrong | Playtest complaints about unnatural entries | Builder supports a max-syllable cap and a denylist file; both are data changes, no code change | - -## Validation Log - -### Session 1 — 2026-09-04 - -**Verification Results** -- Tier: Full (7 phases) -- Claims checked: 14 · Verified: 11 · Failed: 1 · Imprecise: 2 · Unverified (env): 1 - -| Claim | Result | Evidence | -|---|---|---| -| proto `go_package` = `github.com/tiennm99/noitu/server/...` | **FAILED** | `git remote -v` → `github.com/tiennm99dev/noitu.git`. Corrected to `github.com/tiennm99dev/noitu/server` | -| `coder/websocket` provides `SetReadLimit`, `AcceptOptions.OriginPatterns`, `MessageBinary` | VERIFIED | pkg.go.dev, v1.8.15, ISC | -| "Read deadline per frame" via a socket deadline setter | **IMPRECISE** | coder/websocket exposes no `SetReadDeadline`; deadlines are `context.WithTimeout` on `Read`. Phase 5 corrected | -| `dictionary.db` downloadable from `minhqnd/dictionary` Releases | VERIFIED | GitHub API: release `v2.0.0`, asset `dictionary.db` | -| Upstream DB size assumed manageable | **IMPRECISE** | Asset is **179 MB**. Plan had no size claim; Data Distribution section added | -| `@bufbuild/protobuf` / `@bufbuild/protoc-gen-es` current | VERIFIED | npm registry, both v2.14.1 | -| Go ≥1.24, Node, Docker available locally | VERIFIED | go1.26.5, node v24.18.0, Docker 29.6.2 | -| `buf` installed locally | **UNVERIFIED (env gap)** | `buf: command not found`. Not blocking — generated code is committed, so `buf` is needed only to change the schema | - -**Decisions confirmed** - -| # | Question | Decision | Impact | -|---|---|---|---| -| 1 | Go module path | `github.com/tiennm99dev/noitu/server` | Phase 4 `go_package` corrected | -| 2 | Derived DB distribution | Do not republish — every build downloads upstream and derives locally | New Data Distribution section; phase 1 download step, phase 7 CI + Docker strategy | -| 3 | Turn time limit | 20s, identical for bot and PvP | Phase 5 default constant | -| 4 | Deployment target | Docker on a VPS behind a reverse proxy | Phase 7 docs scoped to proxy config, not a PaaS | -| 5 | Word length rule | **≥ 2 syllables** (changed from strict 2) | Phases 1, 2, 3, 4, 6 — filter, API, reject reason, proto enum, UI copy | -| 6 | Player identity in PvP | **User-typed nickname** | Phases 4, 5, 6, 7 — proto field, sanitization, input UI, display | - -**Phase propagation:** phases 1-7 all updated. Reject reason renamed -`NOT_TWO_SYLLABLES` → `TOO_FEW_SYLLABLES` across the engine, proto, and Vietnamese copy. - -### Whole-Plan Consistency Sweep - -Re-read `plan.md` + all 7 phase files after propagation. - -| Check | Result | -|---|---| -| Stale "2-syllable"/"exactly 2" claims | Resolved — all now "≥2 syllables" / "at least 2" | -| `NOT_TWO_SYLLABLES` enum name | Resolved — renamed everywhere (engine, proto, i18n) | -| `go_package` module path | Resolved — single correct value in phase 4 | -| Turn limit stated inconsistently | Resolved — 20s in plan.md + phase 5, `NOITU_TURN_LIMIT` override | -| Nickname referenced but never sourced | Resolved — proto field, sanitization, input, and display now specified | -| `online/+page.svelte` Create vs Modify across phases 6/7 | Resolved — phase 6 creates the stub, phase 7 modifies it | -| Data distribution vs Docker/CI steps | Resolved — consistent across plan.md, phase 1, phase 7 | -| Duplicate embedded proto schema | Single copy, in phase 4 only | - -**Unresolved contradictions: none.** - -## Phase 1 Outcome (2026-09-04) - -Built and verified against the real upstream release. - -| Metric | Value | -|---|---| -| Words | 48,216 (floor 40,000) | -| 2 / 3 / 4+ syllables | ~40k / ~4.5k / ~3.5k | -| Aliases | 2,041 | -| Derived DB | ~3 MB, from a 179 MB source | -| Coverage | `internal/vietnamese` 100%, `cmd/build-dictionary` 82.4% | - -Upstream schema auto-detected (`words.word` / `lang_code`) with no flag override needed, -retiring the phase-1 "unknown upstream schema" risk. Upstream asset pinned by SHA-256. - -**Four defects were found and fixed in alias generation, three of them only visible by -inspecting real output rather than fixture tests:** `quý`→`qúy`, `gì`→`gỳ`, -`hoàn`→`hòan` (tone shift must be confined to open syllables), and an i/y rule so broad it -invented ~340 non-words such as `chức vỵ`. A fifth, opposite defect was found by review: -varying one syllable at a time never produced `hóa lý`, the spelling most people type. -Variants are now generated as a cross product. - -A Vietnamese phonotactic check (closed onset/nucleus/coda inventories) now rejects -loanwords the multilingual source tags as Vietnamese — `credit card`, `world cup`, -`come out`. Its first version wrongly rejected the entire `gì`/`gỉ`/`gìn` family by -greedily matching the `gi` digraph; it backtracks over onset candidates now. - -## Phase 4 Outcome (2026-09-04) - -The wire contract is fixed and generated into both languages. Detail in -[`phase-04`](./phase-04-protobuf-contract-and-codegen.md#phase-4-outcome-2026-09-04). - -Enum values carry buf's mandated enum-name prefix (`REJECT_REASON_TOO_FEW_SYLLABLES`, not -`TOO_FEW_SYLLABLES`), so `buf lint` passes with no carve-out; `protobuf-es` strips the prefix, -leaving the JS side reading `RejectReason.WRONG_LINK`. Two fields were added that the plan -sketch lacked: `PlayedWord.typed`, because the engine keeps the player's raw input alongside -the canonical spelling and the UI has to show a correction happened, and -`REJECT_REASON_GAME_OVER`, because `game.ReasonGameOver` exists and the exhaustiveness test -correctly refused a wire contract without it. - -`web/` now exists as a bare npm package — protobuf and Vitest only — because phase 4's own -JS codegen and cross-language test need somewhere to live. Phase 6 layers SvelteKit on top. - -Both suites read the same 17 binary fixtures in `proto/testdata/`, emitted by the Go tests. -Review caught four guards that were reporting green while protecting nothing — a `buf -breaking` baseline that cannot resolve on a PR checkout, a sync check blind to untracked -files, and two tests whose oracles could drift with the code they checked. All four were -fixed and then negative-tested by deliberately breaking each one. - -## Phase 5 Outcome (2026-09-05) - -The transport layer is in: rooms, turn timers, the bot as a virtual player, reconnect, and -a single binary that serves both the API and the frontend. Detail in -[`phase-05`](./phase-05-go-websocket-server-and-rooms.md#phase-5-outcome-2026-09-05). - -The concurrency contract holds structurally rather than by convention: the room goroutine -starts before anyone is seated and seating itself is a message, so reading `run()` is a -complete proof that one goroutine owns each engine. The bot searches a frozen board rather -than the live engine, and reads carry no deadline — liveness is ping-based, because a read -timeout cannot distinguish a healthy player idling in the lobby from a dead socket. - -Seven defects were found and fixed: three by writing the tests, three more by review, and -a seventh while fixing those. The serious one was authorization — the hub bound a joiner -to a seat before the room decided whether to seat them, so anyone holding a room code -could resign or play on a seated player's behalf. Since the room code is the only -credential online 1v1 has, that was a phase-7 release blocker caught a phase early. - -## Phase 6 Outcome (2026-09-05) - -The vs-bot UI is built and served by the Go binary. Detail in -[`phase-06`](./phase-06-sveltekit-frontend.md#phase-6-outcome-2026-09-05). - -The store is a reducer over `ServerMessage` and computes nothing: validity, turn order and the -result are read from the wire, which is what lets one screen serve both the bot and, in phase -7, online play. The word field is deliberately uncontrolled, because a Telex input method -composes a diacritic over several keystrokes and writing the value back cancels it. The -countdown is drawn against the server's clock and settles 300ms early, so the ring never claims -more time than the server allows. - -Three defects were found by building and five more by review. The serious one was that the game -lifecycle was inferred from the game model rather than owned: the screen asked for a bot game -whenever the board was idle, and clearing the board for a rematch is exactly that condition, so -every rematch started two server-side rooms that then destroyed each other. A request is stored -intent now, with tests over the wiring whose absence let it through. Review also caught a word -silently dropped when typed during a reconnect, a protocol bump that would have put every open -tab into a reconnect loop, an opening turn counted against the device clock, and an -`index.html` cacheable across a deploy. - -Four criteria need a real browser and stay open: one-handed mobile use, Telex diacritic entry, -the absence of a theme flash, and reconnect after the server is killed mid-game. - -## Phase 7 Outcome (2026-09-05) - -Online 1v1 is playable end to end and the game ships as one 24 MB container image. Detail in -[`phase-07`](./phase-07-online-1v1-and-release.md#phase-7-outcome-2026-09-05). - -The rematch needed the one thing phase 5 deliberately did not do: let a room outlive its game. -It now does, but only while both seats hold live connections, so a bot room still closes the -moment its game ends. There is no decline message — leaving is the decline, which the server -already learns from the socket closing. - -Building the browser suite found four defects the unit tests could not: a player who refreshed -mid-game could not rejoin, a restored opponent never cleared the other player's disconnect -banner, a socket that failed without closing was invisible to the client, and a nickname typed -on the online screen never reached the server because the handshake had already gone. - -Two testing tools turned out not to work here and are documented where the next person will -look: offline emulation does not cut a loopback socket, and a routed socket's close does not -reliably reach the server, so the grace window never starts. - -Review then reproduced three more. Two follow from the new state this phase introduced — a room -that outlives its game — and one predates it: a game ending on the turn clock never offered a -rematch although the button was there, a second connection presenting the same resume token -disconnected the player who had not left, and rooms leaked whenever one connection created -several. All three now have regression tests, and the two reachable through the room loop were -negative-tested by reverting the fix. - -## Open Questions - -1. Does the losing player see the words the bot *could* have played (a teaching feature), or just the result? Plan currently assumes just the result. -2. Should the builder cap maximum syllables (e.g. reject 5+ syllable entries as phrases rather than words)? Plan currently applies no upper bound; the cap exists in the builder as a flag if playtesting says otherwise. -3. Domain name / TLS certificate source for the VPS deployment. -4. Should the turn clock pause while a seat is inside its reconnect grace window? As - built the grace window (30s) is longer than the turn limit (20s), so a player who - drops on their own turn always loses on time before the window closes — the grace is - unreachable for the player most likely to need it. Their reconnect is now told - `game_already_over` rather than left waiting, but the underlying policy is unresolved. -5. Should bot rooms be registered in the joinable room namespace at all? They have no code - to share, and excluding them removes a whole class of stranger interference for free. -6. Is closing the connection the right response to a full outbox mid-game? 32 queued - frames is not much on a lossy mobile link, and the opponent currently sees that player - as having left. diff --git a/plans/260907-1551-room-lobby-ready-system/plan.md b/plans/260907-1551-room-lobby-ready-system/plan.md deleted file mode 100644 index f290673..0000000 --- a/plans/260907-1551-room-lobby-ready-system/plan.md +++ /dev/null @@ -1,70 +0,0 @@ -# Room lobby with owner/guest roles and ready-up - -**Status:** delivered - -## Outcome - -An online room is a persistent lobby rather than a wrapper around one game. The -first occupant is the owner, the second is the guest. The guest readies; the -owner starts once they are; a finished game returns both to the lobby, where the -guest must ready again. Players can leave, the owner can kick an unready guest, -and an owner who leaves hands the room to whoever is left. - -## Decisions taken - -- **The owner has no ready flag.** Pressing Start *is* their readiness, so only - the guest can be ready, blocked from leaving, or kicked. -- **Room lifetime:** closes when the last seat empties, or after 10 minutes in - the lobby with no game started. A running game is bounded by the turn clock. -- **The rematch handshake is retired.** `RequestRematch`/`RematchState` stop - being sent and their proto tags are reserved; the lobby is the only place a - next game is negotiated. -- **A kick only frees the seat.** The kicked player may rejoin with the code. - -## Constraints - -- Proto changes are additive; retired tags go to `reserved`, never reused. -- The engine stays two-player; `seats [2]*seat` stays the room's shape. -- Readiness, roles, start and kick are server state. The client renders one - snapshot message and derives nothing. -- Only the room goroutine touches room state; new actions arrive as inputs. -- Every lobby action authorizes against the seat, not the claimed id: a room - code is a shared secret by design. -- Vietnamese copy stays in `web/src/lib/i18n/vi.js`; error codes stay UI keys. - -## Non-goals - -Bot games (start immediately, no readiness), spectators, more than two seats, -kick bans, accounts, and resuming into a room whose game already finished. - -## Acceptance criteria - -1. A guest who joins sees a lobby, not a board. No engine exists until Start. -2. Start is refused unless the guest is seated, connected and ready. -3. Game over returns both to the lobby with the guest unreadied; a second game - needs a fresh ready. -4. Leave while ready is refused; leave while unready frees the seat and the room - survives. -5. Kicking a ready guest is refused; kicking an unready one frees the seat and - tells them why. -6. An owner who leaves, or drops for good, promotes the other player, who is - unreadied and can start once someone joins. -7. The last player out closes the room; an idle lobby closes on the TTL. -8. A reconnect lands back in the lobby with roles and readiness intact. -9. Start racing an unready or a leave, and a kick racing a ready, resolve to an - error for the loser and never a half-started game. - -## Phases - -1. **Wire contract.** `SetReady`, `StartGame`, `KickPlayer`, `LeaveRoom`, - `RoomState`; reserve the rematch tags; regenerate both trees; extend the - cross-language fixtures. -2. **Room lifecycle.** Owner/guest roles, ready flag, start, kick, leave, - promotion, idle TTL, resume into a lobby, retire the rematch path. Route and - rate-limit the new messages in `session.go`. -3. **Client.** A `lobby` phase in the game store driven by `RoomState`, a Lobby - component, retire `RematchPrompt`, copy, and the `/online` route wiring. -4. **Tests.** Room tests for each acceptance criterion, store tests for the new - phase, e2e for join → ready → start → play → back to lobby, leave, kick and - promotion. -5. **Docs.** The online-play section of the README. diff --git a/plans/260907-1636-room-chat/phase-01-start.md b/plans/260907-1636-room-chat/phase-01-start.md deleted file mode 100644 index 64ffedb..0000000 --- a/plans/260907-1636-room-chat/phase-01-start.md +++ /dev/null @@ -1,118 +0,0 @@ ---- -phase: 1 -title: "Wire contract" -status: completed -priority: P1 -effort: "1h" -dependencies: [] ---- - -# Phase 1: Wire contract - -## Overview - -Adds the three messages chat needs to `proto/noitu/v1/game.proto`, regenerates -both trees, and extends the cross-language fixtures so the Go and JavaScript -clients are checked against one artifact. - -## Requirements - -- Functional: a client can send text; the server can deliver one message to - each recipient rendered from their side, and replay a bounded history. -- Non-functional: additive only. The current maxima are `ClientMessage` 12 - (`leave_room`, with 8 reserved) and `ServerMessage` 12 (`room_state`, with - 2, 3 and 11 reserved), so the next free tags are 13 on the client oneof and - 13 and 14 on the server oneof. - -## Architecture - -```proto -// SendChat is one line of text from a seated player to the other. -message SendChat { - string text = 1; -} - -// ChatMessage is one line as one recipient sees it. Rendered per recipient -// like every other room message: from_me is the only field that differs, and -// it is what lets the client style its own words without matching names. -message ChatMessage { - bool from_me = 1; - // Server-sanitized, as everywhere a name is shown. Empty when the author - // has left the room: the seat they spoke from may be somebody else's now, - // and a name outlives neither. The client labels an empty author itself. - string author = 2; - string text = 3; - // Server clock. The client renders it; it never orders by its own clock. - int64 sent_unix_ms = 4; -} - -// ChatHistory is the whole panel, oldest first, sent when a player is seated -// in a room or resumes into one. A snapshot rather than a replayed stream, for -// the same reason RoomState is one: a client that missed frames is correct -// again from the next one instead of having to catch up on events. -// -// Scoped to the recipient: it carries only what was said while they held their -// seat, which is why it is built per seat rather than broadcast. -message ChatHistory { - repeated ChatMessage messages = 1; -} -``` - -Wired into the oneofs as `send_chat = 13` (client) and `chat_message = 13`, -`chat_history = 14` (server). - -`ProtocolVersion` does **not** change, and the honest reason is not that an -old client is unaffected — it is pushed `ChatMessage` frames it never asked -for, because delivery is unconditional and there is no capability negotiation. -It drops them silently: the generated union yields `case: undefined` and the -store's `apply` switch has no `default` arm. The degradation is one-way chat -for the length of a deploy, and it is accepted rather than unnoticed. Phase 3 -adds the `default` arm so the next contract addition is at least visible in a -console. If one-way chat is ever *not* acceptable, bumping `ProtocolVersion` is -the only mechanism the codebase has, and it forces a stale tab to reload. - -## Related Code Files - -- Modify: `proto/noitu/v1/game.proto` -- Modify (generated, do not hand-edit): `server/gen/noitu/v1/game.pb.go`, - `web/src/lib/proto/noitu/v1/game_pb.js`, `web/src/lib/proto/noitu/v1/game_pb.d.ts` -- Modify: `server/internal/wsapi/wire_test.go` — add `client_send_chat`, - `server_chat_message`, `server_chat_history` variants -- Create (generated): `proto/testdata/client_send_chat.bin`, - `proto/testdata/server_chat_message.bin`, `proto/testdata/server_chat_history.bin` -- Modify: `web/tests/game-wire.test.js` - -## Implementation Steps - -1. Add the three messages and their oneof arms to `game.proto`. -2. Lint and regenerate: `go run github.com/bufbuild/buf/cmd/buf@v1.58.0 lint` - then `... generate` from the repo root (no global `buf` needed). -3. Add the three variants to `wire_test.go`. Give them Vietnamese text with - diacritics and a real millisecond timestamp — the JavaScript runtime - surfaces `int64` as a bigint, and a zero value would skip that. Give - `ChatHistory` two messages: one from each side, and one with an empty - `author`, so the repeated field, both `from_me` values and the departed-author - case are all exercised. -4. Regenerate fixtures: `cd server && go test ./internal/wsapi -update`. -5. Assert the new fixture in `web/tests/game-wire.test.js`, including that - `sentUnixMs` arrives as a bigint and that an empty `author` survives the - round trip. - -## Success Criteria - -- [x] `buf lint` passes and both generated trees are current. -- [x] `TestVariantTablesCoverEveryOneofArm` passes, so every new arm is covered. -- [x] `go test ./internal/wsapi` and `npm test` both decode the new fixtures. -- [x] `git diff` on the generated trees shows only the new messages. - -## Risk Assessment - -Low. The one thing to get wrong is a tag collision with a reserved number; -`buf lint` catches it. If a later phase finds `ChatHistory` unnecessary, the -tag is reserved rather than reused — the cost of carrying it is one unused -message, not a migration. - -The `author`-empty convention is load-bearing for Phase 2: if Phase 2 stores -and sends a departed player's name instead, two different people can appear -under one attribution. Signal: the Phase 4 vacate tests. Response: the -convention is stated here and tested there; keep them in step. diff --git a/plans/260907-1636-room-chat/phase-02-room-chat-server.md b/plans/260907-1636-room-chat/phase-02-room-chat-server.md deleted file mode 100644 index 0864b10..0000000 --- a/plans/260907-1636-room-chat/phase-02-room-chat-server.md +++ /dev/null @@ -1,241 +0,0 @@ ---- -phase: 2 -title: "Room chat" -status: completed -priority: P1 -effort: "4h" -dependencies: [1] ---- - -# Phase 2: Room chat - -## Overview - -The server side: authorize the sender's seat, sanitize and cap the text, -rate-limit the sender without endangering the recipient, keep the last 20 -messages per room, deliver each one rendered per recipient, and replay to a -seated player what was said while they held that seat. - -## Requirements - -- Functional: a seated player in a non-bot room can send text to the other - seat; both see it. A seating and a resume replay the recipient's own slice of - the history. History outlives a game and dies with the room. -- Non-functional: bounded memory per room; text sanitized before it is stored; - chat rate-limited on its own budget so talking never costs a move, and sized - so that talking cannot close the other player's session either; chat is not - room activity, so the idle timer still bounds the room. - -## Architecture - -**Sanitizer.** `sanitizeNickname` is close to the filter chat needs — NFC, -control and format characters dropped, whitespace collapsed, rune cap — but two -things separate them. Its cap and its empty-input fallback are nickname policy, -and its character filter admits combining marks: `unicode.IsPrint` counts marks -as printable and NFC does not discard uncomposable ones. At 20 runes that is -cosmetic; at 200 it is a glyph cluster hundreds of lines tall that escapes the -panel and covers the board, and it would sit in the history being replayed. - -Extract the shared body into `sanitizeText(raw string, maxRunes, maxMarks int) -string` in `nickname.go`, returning `""` when nothing survives: - -- NFC, then the existing control/format/non-print filter; -- **new:** drop `unicode.Mn`/`unicode.Me` runes beyond `maxMarks` consecutive - after a base rune (2 is enough for any Vietnamese cluster); -- collapse whitespace with `strings.Fields`, then the rune cap. - -`sanitizeNickname` becomes that call plus the `defaultNickname` fallback and the -`denylist` hook. Its existing tests must pass unchanged — Vietnamese names carry -at most two marks per base rune, so the mark budget is invisible to them. - -Collapsing whitespace also means a chat line cannot contain a newline, so no -message can be made to occupy a column of its own in the panel. - -**Constants** (`room.go`, beside the other room constants): - -```go -// chatHistoryLimit is how many messages a room keeps, and the same window the -// client holds. Enough to catch up on after a reload, few enough that a room -// that lives all day cannot grow. -const chatHistoryLimit = 20 - -// maxChatRunes caps one message. Runes, not bytes: Vietnamese is multi-byte -// and a byte cap would cut it far shorter than a Latin one. -const maxChatRunes = 200 - -// maxChatMarks is how many combining marks may follow one base rune. Two -// covers every Vietnamese cluster; more is a stacking attack, not a word. -const maxChatMarks = 2 -``` - -**Rate limit** (`session.go`): a third bucket beside `submitLimiter` and -`roomLimiter` — `chatLimiter = newBucket(chatsPerSecond, chatBurst, ...)` with -`chatsPerSecond = 1`, `chatBurst = 3`. Its own budget so talking never costs a -move, and sized against the recipient rather than the sender: `outboxCap` is 32 -and a single `Write` can stall for `writeTimeout` (10s), so the existing 5/s of -lobby and move traffic plus 1/s of chat stays comfortably inside the window a -stalled reader can drain. - -**Delivery is best-effort.** `session.send` closes a session whose outbox is -full — correct for game frames, wrong for chat, because it hands one player a -way to disconnect the other into a grace-window abandonment loss. Add -`session.trySend(m)` that takes the same encode path but, on a full outbox, -logs and returns `false` instead of calling `close()`. Chat frames — and only -chat frames — go through it. A dropped chat frame is recoverable: the next -resume replays the history. - -**Input.** `chatInput{sess *session, player game.PlayerID, text string}` on the -room's existing channel. In `dispatch`, gate on `chatLimiter` and `currentRoom()` -exactly as `toRoom` does, and answer a refused enqueue: `if !r.send(chatInput{…}) -{ s.send(errorMsg("busy")) }`, matching `handleSubmit`. Nothing about the room's -own state is read on the session goroutine — `r.strategy` in particular is -room-goroutine-only, so the bot-room refusal belongs in the handler, not here. - -**`handleChat` opens with authorization, like every other room mutation:** - -```go -func (r *room) handleChat(m chatInput) { - // The seat, not the claimed id. A connection the room has already - // retired — kicked, or replaced by a reconnect — can still have a frame - // in flight, and its seat may belong to somebody else by the time the - // room drains it. - if !r.occupies(m.sess, m.player) { - m.sess.send(errorMsg("not_your_seat")) - return - } - if r.strategy != nil { - m.sess.send(errorMsg("not_in_a_room")) - return - } - … -} -``` - -**Storage.** On the room: - -```go -// chat is the room's recent conversation, oldest first, capped at -// chatHistoryLimit. It outlives each game the way the room does. -chat []chatEntry -// chatSeq counts every message the room has accepted, ever. A seat records -// the value it was filled at, which is what scopes a replay to what that -// player was actually present for. -chatSeq uint64 - -type chatEntry struct { - seq uint64 - // author and name are cleared together when the seat is vacated: the - // words stay, the attribution does not. Keeping the name would let the - // next person to request that nickname inherit a stranger's messages, - // since distinguish() only compares against the seat that is occupied. - author game.PlayerID - name string - text string - at time.Time -} -``` - -On `seat`: `chatFrom uint64`, set to `r.chatSeq` when the seat is filled -(`handleCreate`, `handleJoin`, `handleStartBot` for symmetry). A resume does not -touch it — the seat persists, so the player gets their conversation back. - -`vacate` clears both `author` and `name` on that seat's entries. `beginGame` -does **not** touch the history: the room's conversation is not the game's. - -**Idle timer.** `resetIdleTimer()` runs at the tail of every run-loop iteration, -so a chat message would restart the ten-minute lobby window and a room could be -held open forever by typing into it once every nine minutes. Chat is explicitly -not activity: track it per iteration — - -```go -case chatInput: - r.handleChat(m) - // Deliberately no resetTurnTimer and no idle reset: talking is not - // playing, and the room's ten-minute bound has to survive a - // conversation. - idleActivity = false -``` - -— and guard the tail call with it. Skipping the reset leaves the running timer -alone, which is what "the clock keeps ticking" means. - -**Replay.** `chatHistoryFor(s *seat)` sends one `ChatHistory` holding the -entries with `seq >= s.chatFrom`, oldest first. Called from `handleCreate`, -`handleJoin`, and `handleResume` — **before** `handleResume`'s -`if r.inLobby() { return }`, or a lobby refresh (the commonest resume) never -gets one. `handleCreate` sends an empty history on purpose: it is what -overwrites a stale array in a client that has been in another room. - -The history therefore *precedes* that input's `RoomState`, which is broadcast -from the run-loop tail after the handler returns. That is fine and the client -must not depend on the order: `chatHistory` replaces wholesale and reads no -`RoomState`-owned field. Do not claim the opposite anywhere. - -## Related Code Files - -- Modify: `server/internal/wsapi/nickname.go` — extract `sanitizeText` with the - mark budget -- Modify: `server/internal/wsapi/room.go` — constants, `chatEntry`, `chat`, - `chatSeq`, `seat.chatFrom`, `handleChat`, `chatHistoryFor`, `vacate` - attribution clearing, the `chatInput` arm and the idle-activity guard in `run` -- Modify: `server/internal/wsapi/session.go` — `chatLimiter`, `trySend`, the - `ClientMessage_SendChat` arm in `dispatch` -- Modify: `server/internal/wsapi/codec.go` — builders if the room's existing - inline style does not fit - -## Implementation Steps - -1. Extract `sanitizeText(raw, maxRunes, maxMarks)` with the mark budget and - re-express `sanitizeNickname` on top of it. Run the existing nickname tests — - they must not change. -2. Add the constants, `chatEntry`, `chat`, `chatSeq` and `seat.chatFrom`; set - `chatFrom` at every seating site. -3. Add `chatLimiter` and `trySend`; add the `dispatch` arm with the `busy` - answer. -4. Implement `handleChat`: `occupies`, bot-room refusal, sanitize, drop empty, - append with the cap and a rising `chatSeq`, broadcast per recipient via - `trySend`. -5. Add the `chatInput` arm to `run` and the idle-activity guard around - `resetIdleTimer()`. -6. Clear `author` and `name` in `vacate`; leave `beginGame` alone. -7. Add `chatHistoryFor` and call it from `handleCreate`, `handleJoin`, and - `handleResume` before the lobby early return. - -## Success Criteria - -- [x] Two seated players exchange messages in the lobby and mid-game. -- [x] `from_me` is true only for the sender's own copy. -- [x] The 21st message pushes the 1st out. -- [x] A joiner's replay holds nothing said before they were seated; a resuming - player's holds everything they were present for. -- [x] A creator receives an empty history, so a stale client array is replaced. -- [x] A lobby resume replays, not only a mid-game one. -- [x] Text over the cap is truncated at a rune boundary; a 199-mark stack is - truncated; control and format characters never reach the other player. -- [x] A burst is refused with `too_fast` and leaves the move budget intact. -- [x] Sustained chat against a stalled reader drops frames rather than closing - that session. -- [x] A dropped `chatInput` answers `busy`. -- [x] An unseated connection, a connection the room has replaced, and a bot - room are all refused. -- [x] A room with chat traffic and no game still closes on the idle timer. -- [x] After a seat is vacated, that seat's messages carry no author and no - name. - -## Risk Assessment - -The authorization and the attribution rules are the two that bite. Skip -`occupies` and a retired connection speaks as a seat somebody else now holds; -keep the departed player's name and a newcomer can request that nickname and -inherit their messages, because `distinguish` only compares against the seated -opponent. Signal: the Phase 4 replaced-connection and vacate tests. Response: -both are a few lines at a single chokepoint — `handleChat`'s first statement -and `vacate`'s loop — so fix in place. - -The mark budget is a judgement call: two marks per base rune covers Vietnamese, -but it is a rule about text, not a standard. Signal: a legitimate message losing -a mark. Response: raise `maxChatMarks`; do not remove it. - -Memory is bounded by construction (20 entries × 200 runes per room). Room -*lifetime* is bounded by the idle timer only because chat is excluded from it — -that exclusion is the load-bearing part, not the byte count. diff --git a/plans/260907-1636-room-chat/phase-03-client-chat-panel.md b/plans/260907-1636-room-chat/phase-03-client-chat-panel.md deleted file mode 100644 index a58af81..0000000 --- a/plans/260907-1636-room-chat/phase-03-client-chat-panel.md +++ /dev/null @@ -1,177 +0,0 @@ ---- -phase: 3 -title: "Client chat panel" -status: completed -priority: P1 -effort: "4h" -dependencies: [1, 2] ---- - -# Phase 3: Client chat panel - -## Overview - -The panel the players actually use: a message list and an input, driven by the -store, rendered in the lobby and on the board. Expanded in the lobby, collapsed -behind an unread count during a game so it does not crowd a phone screen. - -Depends on Phase 2 as well as Phase 1: every behavioural criterion below needs -a server that accepts `SendChat` and emits `ChatMessage`/`ChatHistory`. - -## Requirements - -- Functional: send a message; see both sides' messages in order with the - author's name, or a "left the room" label where the author is empty; a - replayed history replaces the list wholesale; an unread count while - collapsed; every server refusal visible, including in the lobby. -- Non-functional: text rendered as text, never markup; Enter must not send - mid-composition and the field's value must never be written back; the list - must not steal the page's scroll; the list must not grow without bound. - -## Architecture - -**Store** (`game.svelte.js`). One new field and two new arms: - -```js -/** - * The room's conversation, oldest first, capped at the same window the server - * keeps so the two can never disagree about it. - * - * @type {{ fromMe: boolean, author: string, text: string, atMs: number }[]} - */ -chat: [], -``` - -- `chatMessage` → push one entry (`atMs: Number(value.sentUnixMs)` — the field - is an `int64` and arrives as a bigint, exactly like `deadlineUnixMs`), then - trim the front past `CHAT_WINDOW` (20, mirroring `chatHistoryLimit`). -- `chatHistory` → replace the array wholesale. A snapshot replaces; it never - merges. It also resets the unread count: the player is being resynchronised, - not shown new mail. -- Add a `default:` arm to the `switch` that logs an unhandled payload case, so - the next protocol addition degrades visibly instead of silently (Phase 1). - -**Clearing is the subtle part.** `chat` joins the `kept` set, because a game -starting must not wipe the conversation — but `kept` means "survives -`reset()`", and the online route calls `reset()` on *mount and unmount* as well -(`online/+page.svelte`), so `kept` alone would carry room A's conversation into -room B. Two independent fixes, both cheap: - -1. A `clearChat()` on the store, called in the online route's mount and - unmount effect alongside `reset()` — leaving the screen is leaving the room. -2. Phase 2 sends a `ChatHistory` on every seating including `handleCreate`, so - a stale array is overwritten by the server's truth even if the route forgets. - -`leave()` clears it too, as it clears everything. - -**`ChatPanel.svelte`** — props `{ collapsible?: boolean, onsend: (text: string) => void }`. - -- Message list: `
    ` with `overflow-y: auto` in its own scroll container, - `class:mine={entry.fromMe}`, the author's name shown only on the other side's - messages, `t.chatAuthorLeft` where `author` is empty, and time as `HH:mm` from - `atMs`. -- Scroll: oldest-first, so it scrolls to the **bottom** on a new message, and - only when already near it so it cannot yank a player out of scrollback. Write - this explicitly — `ChainHistory.svelte` is *not* the precedent: it renders - newest-first and scrolls unconditionally to `top: 0`. -- Input: **one-way mirroring only.** `oninput={(e) => (draft = e.currentTarget.value)}` - and never write `draft` back to the element — `WordInput.svelte` is - deliberately uncontrolled because writing the value back cancels Telex - composition and mangles the accent, and a `bind:value` here would reintroduce - exactly that. Keep its `oncompositionstart`/`oncompositionend` submit guard. -- Send is enabled when the draft is non-blank after trimming **and** stripping - zero-width and format characters — the same emptiness the server tests for - after sanitizing, which is what makes the server's silent drop unreachable - rather than merely unlikely. -- Length: check the rune count (`[...draft].length`) in the send handler, not - `maxlength` — `maxlength` counts UTF-16 code units, not runes, and interferes - with composition in some engines. The server truncates anyway; this is only - so the player is not surprised. -- Interpolated with `{entry.text}`, never `{@html}`. -- `collapsible`: a header button toggling the list, showing an unread count. - Unread is client-only state — messages that arrived while collapsed, reset on - expand and on a history replace. Per viewer, so nothing about it goes on the - wire. - -**Error surface.** `Lobby.svelte` renders no `game.state.error` today, and the -online route's error paragraph exists only in the pre-room branch — so a -`too_fast`, `busy` or `not_your_seat` refusal in the lobby is invisible. Render -the store's error inside `ChatPanel` (dismissible, `role="alert"`), which covers -every phase the panel appears in and incidentally gives the lobby's own -refusals — `must_unready_first`, `player_is_ready`, `not_the_owner` — somewhere -to land. - -**Placement** (`online/+page.svelte`): - -- `phase === 'lobby'` → `` under ``. -- `phase === 'playing' | 'over'` → `` - passed to `GameBoard` as a `chat` snippet, rendered below the chain. - `GameBoard` already takes optional snippets (`banner`, `gameOver`), so the - bot route passes none and gets no panel — which is how the non-goal stays a - non-goal rather than a runtime check. - -**Copy** (`vi.js`): `chatTitle: 'Trò chuyện'`, `chatPlaceholder: 'Nhắn tin…'`, -`chatEmpty: 'Chưa có tin nhắn nào.'`, `chatUnread: '{n} tin mới'`, -`chatAuthorLeft: 'Đã rời phòng'`. Reuse `t.submit` for the send button. - -## Related Code Files - -- Create: `web/src/lib/components/ChatPanel.svelte` -- Modify: `web/src/lib/stores/game.svelte.js` — `chat`, two arms, the `default` - arm, `kept`, `clearChat()`, `CHAT_WINDOW` -- Modify: `web/src/lib/ws/messages.js` — `sendChat(text)` builder -- Modify: `web/src/lib/components/GameBoard.svelte` — a `chat` snippet prop -- Modify: `web/src/routes/online/+page.svelte` — `say(text)`, `clearChat()` in - the mount/unmount effect, the panel in both phases -- Modify: `web/src/lib/i18n/vi.js` - -## Implementation Steps - -1. Add the `sendChat` builder. -2. Add `chat`, the two arms with the window trim, the `default` arm, - `clearChat()` and the `kept` entry to the store. Write the store test for - `atMs` being a number here, not in Phase 4 — `initialState()` is typed - `@returns {any}`, so `svelte-check` cannot catch a bigint or a misnamed - field, and the `deadlineUnixMs` precedent is guarded by a test for exactly - that reason. -3. Write `ChatPanel.svelte`: list, then the input with one-way mirroring and the - composition guard, then the collapsible header, then the error line. -4. Add the `chat` snippet to `GameBoard`, render the panel from the online route - in both phases, and call `clearChat()` where the route calls `reset()`. -5. Add the copy. -6. `npm run check`, then `npm test`. - -## Success Criteria - -- [x] A message typed in the lobby appears on both screens, in order. -- [x] Own messages are visually distinct; the other player's carry their name; - a departed author's carry the "left the room" label. -- [x] A reload shows that player's conversation, and the client list matches the - server's window exactly — no duplicate, no drift. -- [x] The list stops growing at the window; the unread badge cannot exceed it - and is reset by a history replace. -- [x] Navigating away and creating a new room shows an empty panel. -- [ ] Enter mid-composition does not send, and a Vietnamese word typed through - Telex arrives whole. *(guard implemented; harness cannot synthesise - composition events — see plan.md, Verification gaps)* -- [x] Send stays disabled for whitespace and for a pasted zero-width character. -- [x] A message containing `x` renders as those characters. -- [x] A `too_fast` refusal is visible in the lobby. -- [x] The panel is collapsed on the board and shows an unread count; the bot - screen has no panel at all. -- [ ] No horizontal page scroll at 360px wide; the list scrolls, the page does - not. *(bounded by CSS, checked by hand rather than asserted)* -- [x] `npm run check` reports zero errors. - -## Risk Assessment - -The composition guard is the one that bites: without it Vietnamese input is -unusable and the bug reads as "chat drops words". The trap is reaching for -`bind:value` to drive the disabled state — that is the write-back the comment in -`WordInput.svelte` warns about. Signal: accents mangle while typing, or the -Phase 4 diacritic case fails. Response: one-way mirroring as specified; if the -disabled state proves fiddly, enable Send unconditionally and let the server -drop the empty message rather than reintroducing a controlled input. - -The unread count is client-only, so if it proves fiddly it can be dropped -without touching the server or the contract. diff --git a/plans/260907-1636-room-chat/phase-04-tests.md b/plans/260907-1636-room-chat/phase-04-tests.md deleted file mode 100644 index 348ed2b..0000000 --- a/plans/260907-1636-room-chat/phase-04-tests.md +++ /dev/null @@ -1,146 +0,0 @@ ---- -phase: 4 -title: "Tests" -status: completed -priority: P1 -effort: "4h" -dependencies: [2, 3] ---- - -# Phase 4: Tests - -## Overview - -One test per success criterion in the plan, at the cheapest layer that can hold -it: Go for the rules, Vitest for the store, Playwright for the two things only -a browser proves — Vietnamese input and the panel not eating the board. - -## Requirements - -- Functional: every criterion in `plan.md` has a test that fails without the - feature. -- Non-functional: no sleeps; wait on state the server produced, as the existing - suites do. - -## Architecture - -**Harness first.** `payloadCase` (`wsapi_test.go`) needs `chat_message` and -`chat_history` arms before anything else — its own comment says a missing arm -makes every `await` time out with nothing to say about why. Add a `say(text)` -helper on `testClient`. Note `silentFor` lives in `regression_test.go`, not -`wsapi_test.go` — same package, so it is usable either way. - -`await` **discards** every frame before the one it wants, so it cannot assert -ordering. Any ordering claim must be checked with `recv()` in sequence — and -per Phase 2 the only true ordering claim is that the history precedes that -input's `RoomState`, which the client is explicitly not allowed to depend on. -Do not write an ordering assertion at all; assert arrival, not sequence. - -Chat is broadcast to both seats, so drain both sides before asserting on a -later frame — the discipline the lobby tests already needed. - -**Go — `server/internal/wsapi/`.** Reuse `pvpLobby` and `pvpRoom`. - -- `TestChatReachesBothSeatsRenderedPerRecipient` — `from_me` true for the - sender, false with the author's name for the other. -- `TestChatWorksInTheLobbyAndInAGame` — one message before Start, one after. -- `TestAJoinerSeesNothingSaidBeforeTheySatDown` — owner talks alone, guest - joins, guest's history is empty. The disclosure boundary. -- `TestChatHistoryIsReplayedOnResumeInTheLobby` — talk in the lobby, drop the - socket, resume, assert the conversation comes back. The commonest refresh, - and the case a hook after `handleResume`'s `inLobby()` early return misses. -- `TestChatHistoryIsReplayedOnResumeMidGame` — the same during a game. -- `TestCreatingARoomReplaysAnEmptyHistory` — what overwrites a stale client - array. -- `TestChatHistoryIsCappedAndOrdered` — 25 messages, the window holds the last - 20, oldest first. -- `TestChatHistorySurvivesAGame` — talk in the lobby, play and finish a game, - the history is still there. -- `TestChatTextIsSanitizedAndCapped` — a zero-width joiner and a bidi override - are gone; a 300-rune Vietnamese message comes back 200 runes, still valid - UTF-8; **a 199-mark stack after one base rune comes back with at most two.** -- `TestEmptyChatIsDroppedWithoutAnError` — send whitespace and a lone - zero-width character; assert silence with `silentFor`, and that a real - message still works afterwards. -- `TestChatIsRateLimitedOnItsOwnBudget` — a burst earns `too_fast`, and a - `SubmitWord` immediately after is still accepted. -- `TestSustainedChatDoesNotCloseTheOpponentsSession` — the recipient's outbox - under pressure drops chat frames rather than being closed. Drive it against a - client that stops reading; assert the session survives and the game does not - end in abandonment. -- `TestChatFromAReplacedConnectionIsRefused` — resume a seat, then send on the - retired socket: `not_your_seat`. The `occupies` guard. -- `TestChatFromAKickedConnectionIsRefused` — kick a guest whose message is - already queued, seat a replacement, assert the message is refused rather than - attributed to the newcomer. -- `TestChatNeedsASeat` — a greeted connection in no room: `not_in_a_room`. -- `TestBotRoomHasNoChat` — refused, like the lobby actions. -- `TestChatDoesNotKeepARoomAlive` — short `IdleFor`; chat throughout; the room - still closes with `room_idle_closed`. -- `TestVacatedSeatKeepsItsWordsButLosesItsAuthor` — guest talks, leaves, a third - player joins: the old message arrives with `from_me` false, an empty author, - and **no name**, checked from the remaining player's side as well as the - newcomer's. - -**Vitest — `web/tests/game-store.test.js`.** - -- `chatMessage` appends in arrival order with `atMs` as a number, not a bigint. -- The list stops at the window: push 25, assert 20 and that the oldest is gone. -- `chatHistory` replaces rather than merges (apply twice, no duplicates) and - resets the unread count. -- `reset()` keeps the conversation, `clearChat()` and `leave()` drop it, and - two rooms in sequence do not carry messages over. - -**Playwright — `web/e2e/pvp-game.spec.js`.** - -- Two players talk in the lobby, both see it; they talk again mid-game. -- A Vietnamese word typed through the composition path arrives whole. Drive it - with `insertCompositionText` via CDP or an IME-style key sequence; if the - harness cannot fake composition, assert instead that a `compositionend`-then- - Enter sequence sends exactly once and note the gap. -- A message containing markup renders as text (`getByText('x')`). -- After a reload the guest still sees their conversation. -- Navigating home and creating a new room shows an empty panel. -- At 360×640 the board has no horizontal scroll with the panel present. - -## Related Code Files - -- Modify: `server/internal/wsapi/wsapi_test.go` — `payloadCase` arms, `say` - helper, the chat tests -- Modify: `web/tests/game-store.test.js` -- Modify: `web/e2e/pvp-game.spec.js`, `web/e2e/helpers.js` (a `chat` locator - group beside `board`) - -## Implementation Steps - -1. Add the `payloadCase` arms and the `say` helper first — without them every - chat `await` fails as a timeout with no diagnosis. -2. Add the Go tests; run `go test ./internal/wsapi -count=2` to catch a - frame-ordering flake early, and `-race` for the goroutine-boundary cases. -3. Add the store tests; `npm test`. -4. Add the e2e cases; run with `PLAYWRIGHT_CHANNEL=chrome` when Playwright's - own Chromium is not installed. -5. Full sweep: `go test ./... -race`, `npm test`, `npm run check`, - `npx playwright test`. - -## Success Criteria - -- [x] Every `plan.md` criterion maps to a named test. -- [x] `go test ./... -count=2 -race` is green. -- [x] `npm test` and `npm run check` are green. -- [x] The whole Playwright suite is green, including the pre-existing cases. -- [x] No test sleeps to wait for chat, and no test asserts frame ordering. - -## Risk Assessment - -Faking IME composition in Playwright may not be reachable with the current -harness. Signal: the composition case cannot be written without CDP. Response: -keep the weaker `compositionend`-then-Enter assertion, state the gap in the -final report, and rely on the unit-level guard — do not delete the case and -call it covered. - -`TestSustainedChatDoesNotCloseTheOpponentsSession` is the awkward one to write: -it needs a client that stops reading without closing its socket. Signal: it -cannot be made deterministic. Response: assert the narrower property — that a -chat frame to a full outbox returns false from `trySend` — as a unit test on the -session, and say so rather than dropping the criterion. diff --git a/plans/260907-1636-room-chat/phase-05-docs.md b/plans/260907-1636-room-chat/phase-05-docs.md deleted file mode 100644 index 74b84ea..0000000 --- a/plans/260907-1636-room-chat/phase-05-docs.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -phase: 5 -title: "Docs" -status: completed -priority: P3 -effort: "30m" -dependencies: [4] ---- - -# Phase 5: Docs - -## Overview - -Chat is user-visible behaviour and a change to the wire contract, so the two -documents that describe those get a paragraph each. Nothing else moves. - -## Requirements - -- Functional: a reader of the README learns that rooms have chat and what its - bounds are; the deployment doc's list of fixed timings stays true — which it - does only because Phase 2 excludes chat from the idle timer. If that decision - is ever reversed, this requirement is the thing that breaks first. -- Non-functional: no new documents, no duplication of what the code already - states — link to the constants rather than restating their values twice. - -## Architecture - -**`README.md`, "Online play".** After the lobby paragraphs, one short -paragraph: the two players can talk, the conversation belongs to the room -rather than to a game, a player is replayed what was said while they held their -seat (so a refresh restores it and a newcomer starts at silence), and text is -sanitized by the same filter that sanitizes nicknames before anyone sees it. -Say plainly that a bot game has no chat. - -**`docs/deployment.md`.** The "one timing is not configurable" paragraph is -still true — say why it survived chat: the idle window measures *no game -started*, and chatting deliberately does not reset it, so a room full of -conversation still closes on schedule. Add the two fixed chat constants -(history window and rune cap) beside it. No new section, and no env var, since -none is being added. - -Check while there: does anything in either document still describe the room as -one game's container? The lobby work already fixed the obvious cases, but chat -belonging to the room is a second claim resting on the same fact. - -## Related Code Files - -- Modify: `README.md` -- Modify: `docs/deployment.md` - -## Implementation Steps - -1. Write the README paragraph in the "Online play" section. -2. Add the constants note and the idle-window clarification to - `docs/deployment.md`. -3. Re-read both sections around the edits for a stale claim about rooms. - -## Success Criteria - -- [x] The README says chat exists, that it is per room, that a replay is scoped - to the recipient's own occupancy, and that bot games have none. -- [x] `docs/deployment.md` states that chat does not reset the idle window, - lists the fixed chat constants, and adds no env var that does not exist. -- [x] No other document contradicts either. - -## Risk Assessment - -Negligible on its own, but this phase is where a reversal of Phase 2's -idle-timer decision would surface as a documentation lie rather than a bug. -Signal: `TestChatDoesNotKeepARoomAlive` removed or skipped. Response: rewrite -the deployment paragraph in the same change, never after it. diff --git a/plans/260907-1636-room-chat/plan.md b/plans/260907-1636-room-chat/plan.md deleted file mode 100644 index 446f78e..0000000 --- a/plans/260907-1636-room-chat/plan.md +++ /dev/null @@ -1,210 +0,0 @@ ---- -title: "Room chat" -description: "Text chat between the two players in an online room, in the lobby and during a game" -status: completed -priority: P2 -effort: "" -tags: [online, wire-contract, frontend] -created: 2026-09-07 ---- - -# Room chat - -## Overview - -The two people in an online room can talk to each other. Chat lives with the -room, not with a game: it works in the lobby while they agree on a game, during -the game, and after it. A player is replayed what was said while they were -sitting in their seat, so a refresh does not wipe their conversation. - -The room already broadcasts per-recipient state (`RoomState`) and already -sanitizes untrusted display text (`sanitizeNickname`). Chat reuses both shapes -rather than inventing new ones. - -## Decisions taken - -- **Scope: the whole room.** One panel, available from the moment a player is - seated — lobby, game, and the lobby a finished game returns to. -- **History: the last 20 messages, scoped to the recipient's own occupancy.** - A seat records where the conversation stood when it was filled, and a replay - starts there. A player who reconnects gets their conversation back in full; a - player who joins a room starts at silence. The room still keeps only 20, so a - long-lived room cannot grow. -- **Abuse controls: server-side only.** A dedicated per-session token bucket, a - rune cap, and the existing sanitizer extended with a combining-mark budget. - No mute button, no filter list. -- **Chat is not room activity.** A chat message does not reset the lobby's idle - timer, so the ten-minute bound the deployment doc promises stays true and a - room cannot be held open indefinitely by typing into it. -- **A message that sanitizes to nothing is dropped without an error, and the - client makes that unreachable** by trimming and stripping zero-width - characters before it will enable Send. Every *other* refusal is answered and - shown. -- **A chat line is best-effort; a history is not.** A full outbox drops a - `ChatMessage` rather than closing the recipient's session — a lost line is - recoverable and closing a session costs the victim the game. `ChatHistory` - still goes through `send`, because it is the frame that corrects a whole - panel and there is nothing behind it. -- **A replayed history counts as what it holds, not as nothing.** A resync - keeps a folded panel's unread count honest instead of stranding it. -- **A vacated seat's messages keep their text and lose their author entirely** — - no id and no name — so neither the next occupant nor the player who stayed can - have a stranger's words attributed to them. - -## Constraints - -- Proto changes are additive; retired tags go to `reserved`, never reused. -- Chat text is untrusted input rendered in another player's browser. It passes - the sanitizer before it is stored or sent, and is rendered as text, never as - markup. -- Only the room goroutine touches room state; chat arrives as another input, - and every chat action authorizes with `occupies` on that goroutine — a room - code is a shared secret by design, and a connection the room has already - retired must not be able to speak as its old seat. -- Vietnamese copy stays in `web/src/lib/i18n/vi.js`; error codes stay UI keys. -- The chat input is typed with a Telex or VNI input method, so Enter must not - send mid-composition, and the field's value must never be written back into - the DOM node — see `WordInput.svelte`'s comment for why. - -## Non-goals - -Bot games (nobody to talk to), spectators, emotes or canned phrases, message -editing or deletion, typing indicators, read receipts, chat that outlives the -room, a mute or report control, and any profanity list. - -## Goals - -| # | Goal | Priority | -|---|------|----------| -| 1 | Two seated players can exchange text in the lobby and during a game | P1 | -| 2 | A reconnect replays what that player was present for | P1 | -| 3 | Text is capped, rate-limited and sanitized before anyone sees it | P1 | -| 4 | Chat cannot cost a player their session, their room, or their game | P1 | -| 5 | The panel does not crowd the board on a phone | P2 | - -## Phases - -| # | Phase | Status | -|---|-------|--------| -| 1 | [Phase 1: Wire contract](./phase-01-start.md) | Done | -| 2 | [Phase 2: Room chat](./phase-02-room-chat-server.md) | Done | -| 3 | [Phase 3: Client chat panel](./phase-03-client-chat-panel.md) | Done | -| 4 | [Phase 4: Tests](./phase-04-tests.md) | Done | -| 5 | [Phase 5: Docs](./phase-05-docs.md) | Done | - -Phases run in order. 2 depends on 1; 3 depends on 1 and 2; 4 depends on 2 and 3. - -## Success Criteria - -- [x] A seated player can send a message and the other player sees it, in the - lobby and during a game. -- [x] Each side sees its own messages marked as its own, and the author's - server-sanitized name on the other's. -- [x] A player who reloads is shown their conversation again; a player who - joins a room is shown nothing said before they sat down. -- [x] History survives a game start and a game end; it dies with the room. -- [x] The client's list and the server's history hold the same 20 messages - after a replay, and the unread badge cannot outlive them. -- [x] A message longer than the cap is truncated at a rune boundary, and a - stack of combining marks is truncated with it. -- [x] Control and format characters cannot reach another player's screen, and - no message is rendered as markup. -- [ ] Every refusal is answered and visible: `too_fast` on a burst, `busy` on a - dropped input, `not_your_seat` from a retired connection — including in - the lobby, which today renders no error at all. -- [x] A burst does not consume the sender's move budget, and sustained chat - does not close the recipient's session. -- [x] Chat from a connection with no seat, from a connection the room has - replaced, and from a bot room are all refused. -- [x] A room with chat traffic and no game still closes on the idle timer. -- [x] A seat that is vacated leaves its messages with no author at all, so - neither player can be shown a stranger's words as somebody's. -- [ ] Enter mid-composition does not send a half-typed Vietnamese word. - -## Verification gaps - -Everything above is implemented. These paths are implemented but not proven by -a test, and are listed here rather than ticked: - -- **Enter mid-composition** (criterion 13). The guard is `oncompositionstart` / - `oncompositionend` around a submit that reads the element, identical in shape - to `WordInput.svelte`'s. Playwright in this harness cannot synthesise - composition events, which the plan anticipated; covered by review only. -- **`busy` on a dropped input** (part of criterion 8). Filling a 32-slot room - inbox deterministically is not reachable from a test client. `too_fast` is - covered on both sides — refused on the server, visible in the lobby panel — - and `not_your_seat` is covered on the server. -- **`vacate`'s kick and grace-expiry paths.** The re-sync is asserted for a - voluntary leave only; the other two callers were traced by reading. -- **The departed-author label.** The server clears attribution and two tests - assert it; nothing asserts the client renders "Đã rời phòng" for it. -- **`ChatPanel.svelte` has no component test**, because the repo has no Svelte - component suite. Everything in that file is covered end-to-end or not at all. -- **The 360px board layout** (goal 5). Bounded by `max-height` and - `overflow-wrap: anywhere`, checked by hand rather than asserted. - -## Risks - -- **The panel eats the board on a phone.** Mitigation: collapsed by default - during a game with an unread count; the lobby shows it expanded. Signal: the - e2e board assertions start needing scrolling. Response: make it a sheet. -- **The unread badge is client-only state derived from a server counter.** Two - reviewers found the same defect in it by reading, twice: first counting a - capped list's length, then going quiet after a resync reset the counter under - it. It is now `chatCount` set to the replayed length and a `seenAt` that - follows a decrease. Signal: a badge that stops counting, or one that counts - what was already read. Response: the arithmetic is four lines in one - component — but add the store test before the fix, not after. -- **A client applies `ChatHistory` before it has a room.** The history is sent - from the handler and therefore *precedes* that input's `RoomState`, which is - broadcast from the run-loop tail. The client must not depend on the order — - `chatHistory` replaces wholesale and reads no `RoomState`-owned field. - Signal: a panel rendering with an empty room code. Response: render the panel - from `phase`, never from the presence of messages. -- **The combining-mark budget is a judgement call, not a standard.** Signal: a - legitimate Vietnamese message loses a mark it needed. Response: the budget is - one constant in one function — raise it; do not remove the cap. - -## Red Team Review - -### Session — 2026-09-07 -**Findings:** 15 distinct, deduplicated from 28 raw across 3 reviewers -(15 accepted, 0 rejected) -**Severity breakdown:** 3 Critical, 7 High, 5 Medium -**Reviewers:** Security Adversary (Fact Checker), Failure Mode Analyst (Flow -Tracer), Assumption Destroyer (Scope Auditor). Verification tier: Full. -**Evidence filter:** every finding carried `file:line` citations; none rejected -for want of evidence. - -| # | Finding | Severity | Disposition | Applied To | -|---|---------|----------|-------------|------------| -| 1 | `chat` in the store's `kept` set leaks one room's conversation into the next; `reset()` also runs on route mount/unmount, and `handleCreate` sends no history to heal it | Critical | Accept | Phase 2, 3, 4 | -| 2 | Replay ordering inverted — `broadcastRoomState` runs in the run-loop tail — and `handleResume` early-returns for a lobby resume, so the commonest refresh never replays | Critical | Accept | Phase 2, 4, plan.md | -| 3 | `handleChat` specified without `occupies`; the claim that `toRoom` does a seat check is false. Bot-room check cannot live in `dispatch` — `r.strategy` is room-goroutine-only | Critical | Accept | Phase 2, 4 | -| 4 | Chat resets the lobby idle timer, so a room can be held open forever and Phase 5's "timings stay true" is unsatisfiable | High | Accept | Phase 2, 4, 5 | -| 5 | A separate chat budget is additive to the existing ones, and a full outbox closes the recipient's session — sustained chat can disconnect an opponent into an abandonment loss | High | Accept | Phase 2, 4 | -| 6 | "Empty message is unreachable" is false — spaces or a pasted zero-width character pass the blank check | High | Accept (user decision: fix the client's blank check, keep the silent drop) | Phase 3, 4, plan.md | -| 7 | The 20-message replay discloses a private conversation to any stranger who redeems the code, and lets an owner pre-load a payload for joiners | High | Accept (user decision: replay scoped to the recipient's own occupancy) | Phase 2, 4, plan.md | -| 8 | The sanitizer drops control and format characters but not combining marks; at 200 runes that is a render bomb that persists in history | High | Accept | Phase 2, 4 | -| 9 | A dropped `chatInput` answers nothing, unlike every other input; and the lobby renders no error surface at all | High | Accept | Phase 2, 3, 4 | -| 10 | A controlled chat input cannot copy `WordInput`'s deliberately-uncontrolled composition guard verbatim; `maxlength` counts UTF-16 units, not runes | High | Accept | Phase 3 | -| 11 | The client array is unbounded (the cap is server history only) and the unread badge diverges from the list on a replace | Medium | Accept | Phase 3, 4 | -| 12 | Keeping a departed player's `name` lets a newcomer take that nickname — `distinguish` only checks the seated opponent — and contradicts Phase 1's own proto comment | Medium | Accept | Phase 1, 2, 4 | -| 13 | The `ProtocolVersion` justification is false: chat is pushed to old clients, which drop it silently for want of a `default:` arm | Medium | Accept | Phase 1, 3 | -| 14 | Phase 3's success criteria mostly require Phase 2's server, but its dependencies said `[1]` | Medium | Accept | Phase 3, plan.md | -| 15 | Four false citations: `ChainHistory` scrolls to top unconditionally; tag maxima are 12/12; `silentFor` lives in `regression_test.go`; `payloadCase` needs new arms — plus "JSDoc keeps the panel honest" when the store's state is typed `any` | Medium | Accept | Phase 1, 3, 4 | - -### Whole-Plan Consistency Sweep -- Files reread: plan.md, phase-01-start.md, phase-02-room-chat-server.md, - phase-03-client-chat-panel.md, phase-04-tests.md, phase-05-docs.md -- Decision deltas checked: 7 — occupancy-scoped replay; chat excluded from idle - activity; best-effort chat delivery; `occupies` on every chat input; author - cleared entirely on vacate; client blank check strict; client list capped to - the server's window -- Reconciled stale references: 9 — the replay-ordering claim in plan.md Risks - and Phase 2; "last 20 on join" in plan.md Decisions, Goals and Phase 4; the - `toRoom` seat-check claim in Phase 2; the `ProtocolVersion` justification and - tag maxima in Phase 1; the `name`-retention rule in Phase 2 against Phase 1's - proto comment; the `ChainHistory` scroll precedent and the JSDoc claim in - Phase 3; Phase 3's dependency list; Phase 5's "timings stay true" -- Unresolved contradictions: 0 diff --git a/plans/260908-1006-multiplayer-rooms/phase-01-engine-elimination.md b/plans/260908-1006-multiplayer-rooms/phase-01-engine-elimination.md deleted file mode 100644 index 612c27b..0000000 --- a/plans/260908-1006-multiplayer-rooms/phase-01-engine-elimination.md +++ /dev/null @@ -1,48 +0,0 @@ -# Phase 1 — Engine: elimination and standings - -`server/internal/game/engine.go`, `state.go`. - -The engine already carries `players []PlayerID` and advances `turnIndex` modulo -its length, so seating more than two costs nothing. What is two-player is the -*ending*: `opponentOf` picks the winner, and any single failure finishes the -game. - -## What changes - -- `alive []bool` parallel to `players`, plus `aliveCount`. -- `outOrder []PlayerID` — elimination order, first out first. Rank is derived - from it, so nothing has to be recomputed on read. -- `outReason map[PlayerID]EndReason` — why each player went out. -- `Turn()` returns the next **living** player; `advance()` skips the dead. -- `eliminate(p, reason)` marks a player out, appends to `outOrder`, and finishes - the game when one player is left. -- `expire(now)` eliminates the player to act, then keeps eliminating the next to - act while the position has no legal move — which is what stops a dead end from - costing every remaining player a full turn clock each. -- `NoMove()` and `Resign(p)` route through `eliminate` too. `Resign` works out - of turn: a player may give up while somebody else is thinking. -- `Alive(p) bool` and `Standings() []Standing` for the transport layer. -- Delete `opponentOf`. - -```go -// Standing is one player's final placing. Rank 1 is the winner; a player -// eliminated later ranks above one eliminated earlier. -type Standing struct { - Player PlayerID - Score int - Rank int - Reason EndReason // EndNone for the winner -} -``` - -`State` gains `Standings []Standing` and `Alive map[PlayerID]bool`. - -## Invariants worth a test - -- Two players: every existing outcome is identical to today's. -- Four players: A times out → A out, turn passes to B on the **same syllable**, - scores and used set survive. -- A dead end closed by A eliminates B, C and D in one call and leaves A standing. -- `Resign` out of turn eliminates the resigner and does not disturb whose turn - it is, unless the resigner *was* the player to act. -- Rank ordering: survivor 1, then reverse `outOrder`. diff --git a/plans/260908-1006-multiplayer-rooms/phase-02-wire-contract.md b/plans/260908-1006-multiplayer-rooms/phase-02-wire-contract.md deleted file mode 100644 index 8625181..0000000 --- a/plans/260908-1006-multiplayer-rooms/phase-02-wire-contract.md +++ /dev/null @@ -1,61 +0,0 @@ -# Phase 2 — Wire contract and codegen - -`proto/noitu/v1/game.proto`, regenerated into `server/gen/` and -`web/src/lib/proto/`. `ProtocolVersion` → **2**. - -## New messages - -```proto -// PlayerSlot is one seat in the room, rendered for one recipient. -message PlayerSlot { - string player_id = 1; // "p1".."p4" — stable within a room - string name = 2; - bool is_me = 3; - bool is_owner = 4; - bool ready = 5; - bool connected = 6; -} - -// PlayerScore is one player in a running or finished game. -message PlayerScore { - string player_id = 1; - string name = 2; - bool is_me = 3; - uint32 score = 4; - bool eliminated = 5; - bool connected = 6; - uint32 rank = 7; // final placing, 1 = winner; 0 while in play -} - -// PlayerEliminated is one player leaving a game that is still running. -message PlayerEliminated { - string player_id = 1; - string name = 2; - bool is_me = 3; - GameEndReason reason = 4; - // What the position still had, sent only to the player who went out. - repeated string suggestions = 5; -} -``` - -## Changed messages - -| Message | Change | -|---|---| -| `RoomState` | `reserved 2, 4, 5, 6, 7, 8` (`i_am_owner`, `i_am_ready`, `opponent_*`). Adds `repeated PlayerSlot players = 9`, `uint32 max_players = 10`, `uint32 min_players = 11`, `uint32 grace_ms = 12`. `room_code` and `can_start` stay. | -| `GameStarted` | Adds `repeated PlayerScore players = 7`, `string turn_player_id = 8`. | -| `TurnUpdate` | `reserved 6, 7` (`my_score`, `opponent_score`). Adds `repeated PlayerScore players = 9`, `string turn_player_id = 10`. `played` may now be absent — an elimination advances the turn without a word. | -| `GameOver` | `reserved 3` (`my_score`). Adds `repeated PlayerScore standings = 6`, ordered by rank. | -| `PlayedWord` | Adds `string player_id = 6`, so a four-way chain says who played what without matching names. | -| `KickPlayer` | Adds `string player_id = 1` — the owner now names a seat. | -| `OpponentLeft` | Retired. `ServerMessage` reserves tag 8; presence travels on `RoomState.players[].connected` with `RoomState.grace_ms`, which is broadcast mid-game already. | -| `ServerMessage` | `reserved 2, 3, 8, 11`; adds `PlayerEliminated player_eliminated = 15`. | - -`i_am_owner` / `i_am_ready` go because the recipient's own row already carries -them: two encodings of one fact are two ways for a client to disagree with the -server. - -## Regeneration - -`buf` is not on this machine — `go install github.com/bufbuild/buf/cmd/buf@latest` -first, then `make proto`. Both generated trees are committed. diff --git a/plans/260908-1006-multiplayer-rooms/phase-03-room-seats.md b/plans/260908-1006-multiplayer-rooms/phase-03-room-seats.md deleted file mode 100644 index 776f0e8..0000000 --- a/plans/260908-1006-multiplayer-rooms/phase-03-room-seats.md +++ /dev/null @@ -1,60 +0,0 @@ -# Phase 3 — Room: N seats, N grace windows - -`server/internal/wsapi/room.go`, `codec.go`, `hub.go`, `session.go`, -`nickname.go`. - -## Seats - -- `maxPlayers = 4`, `minPlayers = 2`, `seatIDs = [maxPlayers]game.PlayerID{"p1".."p4"}`. -- `seats [maxPlayers]*seat`. `freeSeat`, `occupied`, `seatOf` already loop. -- `distinguish(name, taken)` becomes `distinguish(name, taken []string)` and - counts up (" 2", " 3", …) until the name is unused. -- Drop `otherNickname` / `opponentSeat`; add `seatsInOrder()` yielding the - occupied seats in seat order, which is also engine turn order. - -## Lobby - -- `canStart()` — no bot strategy, in lobby, at least `minPlayers` seated, owner - connected, every non-owner seat connected and ready. -- `lobbyStart` refuses with `need_more_players` / `player_offline` / - `not_everyone_ready`. -- `lobbyKick` reads `KickPlayer.player_id`: refuse `no_one_to_kick` when the - seat is empty, `cannot_kick_self` when it is the owner's own, and - `player_is_ready` when that player is ready — the existing rule, per target. -- `handleJoin` refuses a full room exactly as now. - -## Grace windows - -One timer is no longer enough: any number of seats can be inside their window at -once. Each `seat` gains `graceUntil time.Time`; `resetGraceTimer()` arms a single -timer for the earliest of them, and on fire every seat past its deadline is -expired together. That keeps one timer and one wakeup regardless of how many -players dropped. - -A dropped seat mid-game is **not** skipped: the engine's turn clock runs for it, -so a player who drops on their own turn is eliminated by timeout before their -window is up. The window only decides whether they are still in the game after it. - -## Eliminations - -`applyEliminations(before int)` — compare the engine's `outOrder` length against -what it was, send each newly eliminated player's `PlayerEliminated` to every -seat (with `suggestions` only in the copy that goes to the player who went out), -then: - -- game over → `broadcastGameOver` with standings; -- otherwise → `r.turnSeq++` and `broadcastTurn(nil)`, so every client learns the - new deadline and whose turn it is without a word having been played. - -The room keeps `outWire map[game.PlayerID]noituv1.GameEndReason`, because -`OPPONENT_LEFT` is a transport fact the engine never learns: a grace expiry -eliminates with `EndResigned` and is reported as `OPPONENT_LEFT`. - -`GameOver.reason` stays the game-level reason — the reason the **last** -elimination happened — which keeps 1v1 wording identical. - -## Broadcasts - -- `broadcastRoomState` builds `PlayerSlot` rows in seat order. -- `broadcastTurn(move *game.Move)` and `sendGameStarted` build `PlayerScore` rows. -- `gameOverFor` builds standings once and sets `is_me` per recipient. diff --git a/plans/260908-1006-multiplayer-rooms/phase-04-client.md b/plans/260908-1006-multiplayer-rooms/phase-04-client.md deleted file mode 100644 index 0c6f255..0000000 --- a/plans/260908-1006-multiplayer-rooms/phase-04-client.md +++ /dev/null @@ -1,37 +0,0 @@ -# Phase 4 — Client: the room as a list of players - -## Store — `web/src/lib/stores/game.svelte.js` - -`opponentName` / `opponentScore` / `opponentPresent` / `opponentReady` / -`opponentConnected` / `isOwner` / `isReady` / `opponentLeft` all collapse into: - -```js -/** @type {PlayerSlot[]} */ roomPlayers: [], // from RoomState -/** @type {PlayerScore[]} */ gamePlayers: [], // from GameStarted / TurnUpdate -/** @type {PlayerScore[]} */ standings: [], // from GameOver -maxPlayers: 4, -minPlayers: 2, -graceMs: 0, -turnPlayerId: '', -/** @type {{ playerId, name, isMe, reason, suggestions } | null} */ -lastElimination: null, -``` - -with derived getters `me`, `isOwner`, `isReady`, `myScore`, `iAmEliminated`. -A `turnUpdate` with no `played` is an elimination advancing the turn — push -nothing to the chain, apply the rest. - -## Components - -| File | Change | -|---|---| -| `Lobby.svelte` | Renders `roomPlayers` plus empty slots up to `maxPlayers`. Kick button per row, on any guest who is not ready. Hints keyed off how many are seated and how many are ready. | -| `ScoreBoard.svelte` | A row per player, active row = `turnPlayerId`, eliminated rows struck through. Replaces the two-sided layout. | -| `PlayerStatus.svelte` | Replaces `OpponentStatus.svelte`: one line per disconnected player counting down `graceMs`, plus an "eliminated" banner for this player. | -| `GameOverPanel.svelte` | Standings table from `standings`; `myScore` read from the row where `isMe`. Suggestions still only for a player who did not win. | -| `ChainHistory.svelte` | Labels each entry with its player's name via `playerId`. | -| `GameBoard.svelte` | Drops `opponentLabel`; word input hidden while `iAmEliminated`. | -| `online/+page.svelte` | `kick(playerId)`. | -| `messages.js` | `kickPlayer(playerId)`, `PROTOCOL_VERSION = 2`. | -| `history-export.js` | Names each mover from the player list rather than "you / opponent". | -| `i18n/vi.js` | New keys: standings, eliminated, spectating, per-player disconnect, `need_more_players`, `cannot_kick_self`, `player_offline`. | diff --git a/plans/260908-1006-multiplayer-rooms/phase-05-tests-docs.md b/plans/260908-1006-multiplayer-rooms/phase-05-tests-docs.md deleted file mode 100644 index 90ea242..0000000 --- a/plans/260908-1006-multiplayer-rooms/phase-05-tests-docs.md +++ /dev/null @@ -1,30 +0,0 @@ -# Phase 5 — Tests and docs - -## Go - -- `game/engine_test.go` — the phase 1 invariants: four-player elimination order, - dead-end cascade, out-of-turn resign, standings ranks, and every existing - two-player case unchanged. -- `wsapi/wsapi_test.go` — a four-player room: join to full, fifth refused; start - refused until all ready; kick by id; a full game with two eliminations; an - eliminated player still receiving turn updates and chat. -- `wsapi/regression_test.go` — reconnect inside and outside the window with more - than two players seated. -- `wsapi/wire_test.go` — fixtures for the new and changed messages; - `go test ./internal/wsapi -update` rewrites `proto/testdata/`. -- `wsapi/convert_test.go` — unchanged mappings still exhaustive. - -## JavaScript - -- `tests/game-store.test.js` — the new projection, including a `turnUpdate` with - no `played`. -- `tests/game-wire.test.js` — decodes the regenerated fixtures. -- `tests/error-codes.test.js` — every server code has Vietnamese copy. -- `e2e/pvp-game.spec.js` — a three-player game through the browser. -- `e2e/reconnect.spec.js` — unchanged 1v1 path still passes. - -## Docs - -- `README.md`: "online 1v1 by room code" becomes rooms of two to four, and the - elimination rule joins the opening description. -- `docs/deployment.md` only if a knob changed — it did not. diff --git a/plans/260908-1006-multiplayer-rooms/plan.md b/plans/260908-1006-multiplayer-rooms/plan.md deleted file mode 100644 index 81582de..0000000 --- a/plans/260908-1006-multiplayer-rooms/plan.md +++ /dev/null @@ -1,55 +0,0 @@ -# Rooms of two to four players - -Status: **done** — 2026-09-08. Phases 1–5 delivered and verified, except the -Playwright suite, which could not run on this machine: the browser download -times out here. The specs are written and parse; they have not been executed. - -A room seats up to `maxPlayers` (4, one constant) instead of exactly two. The -owner starts once every guest is ready; two seated players is enough. A player -who fails their turn is eliminated and the rest play on, last standing wins. - -## The rules this settles - -| Question | Answer | -|---|---| -| Room size | 2–4 seats. `maxPlayers` is one server constant, sent to the client in `RoomState` so the lobby renders whatever the server allows. | -| Starting | Owner only, and only when every other seated player is connected and ready. Two players is the minimum. | -| Kicking | Owner only, names a seat, refused when that player is ready — the existing rule, now per-target. | -| Failing a turn | The player who runs out of clock is **eliminated**; the syllable and the used set survive, the turn passes to the next living player. Last standing wins. | -| A dead-end position | The player to act still burns their clock and goes out, as today. Everyone after them is then eliminated at once rather than each waiting out a turn they cannot answer — which leaves the player who closed the position standing, exactly the 1v1 outcome. | -| Disconnecting mid-game | The seat's clock keeps running, so a player who drops on their own turn can time out and go out normally. Otherwise their reconnect window decides it: come back and play on, or be eliminated when it expires. | -| Being eliminated | They keep their seat and watch: live chain, live turns, chat still works, no word input. Everyone lands back in the same lobby when the game ends. | -| Game over | Full standings — every player, their score, ranked by finishing order (last standing first, then reverse elimination order). | - -## Wire contract - -This breaks the two-player shape of `RoomState`, `TurnUpdate`, `GameOver` and -`OpponentLeft`, so `ProtocolVersion` goes to **2** and the retired field tags go -to `reserved`. That is what the version number is for: an old client is refused -with a readable error rather than decoding a frame that means something else now. - -## Phases - -| # | Phase | Depends on | -|---|---|---| -| 1 | [Engine: elimination and standings](./phase-01-engine-elimination.md) | — | -| 2 | [Wire contract and codegen](./phase-02-wire-contract.md) | — | -| 3 | [Room: N seats, N grace windows](./phase-03-room-seats.md) | 1, 2 | -| 4 | [Client: the room as a list of players](./phase-04-client.md) | 2 | -| 5 | [Tests and docs](./phase-05-tests-docs.md) | 1–4 | - -All five are complete. `go vet`, `go test ./... -race` and `npm test` pass; -`npm run check` reports no problems. `npx playwright test` is blocked on the -browser download, not on the code. - -## Acceptance criteria - -- A room seats 2–4; a fifth join is refused with `room_full`. -- Owner's start button is refused until every other seated player is ready. -- The owner can kick any named guest who is not ready. -- A 4-player game eliminates on timeout and continues; the last player standing - wins and everyone sees the same standings. -- An eliminated player still sees the board and can chat, and returns to the - lobby with the others. -- 1v1 behaviour is unchanged end to end, including reconnect and vs-bot. -- `go vet`, `go test -race`, `npm test` and the Playwright suite all pass. diff --git a/plans/260908-1525-dictionary-corpus-switch/audit-proper-noun-drops.md b/plans/260908-1525-dictionary-corpus-switch/audit-proper-noun-drops.md deleted file mode 100644 index bfaea77..0000000 --- a/plans/260908-1525-dictionary-corpus-switch/audit-proper-noun-drops.md +++ /dev/null @@ -1,184 +0,0 @@ -# Audit: proper-noun drops and the wiktionary-only corpus - -Date: 2026-09-08. Phase 3 of `plan.md`. Every number here is reproducible from the -commands listed; the shipped `data/noitu.db` was copied aside before any build and is the -"old" side of every comparison. - -## Commands - -```sh -# from server/, after `make fetch-dict` -go run ./cmd/build-dictionary --merged ../data/undertheseanlp-words.jsonl \ - --out /tmp/wik.db --report-drops /tmp/drops-wik.txt -go run ./cmd/build-dictionary --merged ../data/undertheseanlp-words.jsonl \ - --sources hongocduc,wiktionary --out /tmp/hw.db --report-drops /tmp/drops-hw.txt -# sample: python, random.seed(1), random.sample(sorted(drops-wik), 100) -# casualties: (drops-wik − drops-hw) ∩ words(hw.db) -go test ./internal/bot/ -run RealCorpus -v # once per database at data/noitu.db -``` - -## Build result - -``` -rejected 97: contains punctuation -rejected 5074: fewer than 2 syllables -rejected 37: no Vietnamese letters -rejected 4747: only ever capitalized -accepted 22419 distinct words (sources: wiktionary) -``` - -22,419 vs the 22,310 the research report measured. The delta is the rule: the report -dropped any form containing an uppercase letter, the builder drops forms that *begin* with -one, so `tia X`, `bộ bài tây`, `chữ Hán`, `châu Á` survive. All are ordinary words. - -## 1. Drop audit — 100 sampled, seed 1 - -Verdicts: **P** proper noun (correct drop) · **X** not a Vietnamese word (would be rejected -by the filter anyway) · **U** unclear · **C** common word (false positive). - -| | | | | | -|---|---|---|---|---| -| a yun P | an hoà tây P | an minh bắc P | an thuận P | ba vinh P | -| ban cơ U | brao X | bàn đạt P | bành trạch P | bàu năng P | -| bá xuyên P | bình hiệp P | bình tấn P | bằng la P | chiềng sơ P | -| châu hội P | chù lá phù lá P | chư krêy P | cur X | côn đảo P | -| cơ kiều P | cẩm trung P | hùng vương P | hồng bàng P | irc X | -| khánh gia P | lão quân P | lương đài P | mặc dương P | nhơn hoà lập P | -| ninh lai P | ninh thạnh P | noong bua P | nội thôn U | phiếu hữu mai U | -| phù lảng P | phương cao kén ngựa U | phần lão U | quán hành P | quắc hương P | -| rã bản P | sam mứn P | sơn hạ P | sơn vy P | tam ngọc P | -| thanh liên P | thanh văn P | thiệu đô P | thái an P | thượng lâm P | -| thượng tiến P | thạch xuân P | thới quản P | tiên cát P | tiên thọ P | -| tri lễ P | triệu giang P | triệu việt vương P | trung tú P | trà khê P | -| trà kót P | trà linh P | trường khánh P | trường long P | trần đình thâm P | -| trọng do P | tuân lộ P | tân hưng P | tân mỹ P | tân nhựt P | -| tân trì P | tây hiếu P | tăng sâm P | tĩnh gia P | tịnh an P | -| tốt động P | vinh an P | việt–mường X | vân canh P | võ trường toản P | -| võ văn tồn P | văn môn P | văn quân P | văn đình dận P | vĩnh hoà hưng bắc P | -| vĩnh lương P | vĩnh lộc b P | vĩnh trinh P | vũ thư P | vạn thuỷ P | -| xuân mai P | xuân quan P | xá bung P | xá cẩu P | xá khắc P | -| xốp cộp P | yên cát P | ô qua U | đồng nai P | đổ rượu ra sông thết quân lính C | - -**P 89 · X 4 · U 6 · C 1.** Gate is ≤2 common words in 100: **passes.** The bulk is -commune names, ethnonyms and historical persons — the content the rule exists to remove. - -## 2. Case casualties — 215, listed in full - -Words the wiktionary rows hold only capitalized, which another branch has lowercase. They -are dropped under the plan's wiktionary-only evidence rule. Full list: -`case-casualties.txt` in the session scratchpad; reproduced here because it is the -decision input. - -an bình · an dân · an hảo · an khang · an lạc · ba tiêu · biển hồ · bu lu · **bàn là** · -**báo đáp** · bát tiên · bình chuẩn · bình chân · bình khang · bình nghị · bình sa · bình -thanh · bình thuỷ · bình trị · bình tâm · bình văn · bình điền · bích đào · bóng chim tăm -cá · bông trang · bạch cung · bản nguyên · **bảo toàn** · bắc phong · bẻ quế · bố chính · -**bồ đề** · bồng sơn · cao biền dậy non · cao kỳ · cao nhân · cao sơn · cao tổ · cao xanh · -cao đường · chà và · chày sương · chánh hội · **chân mây** · châu lệ · châu thành · chí -thiện · chí thành · chính tâm · **chúa nhật** · chúa trời · chị hằng · con tạo · cát lũy · -cát tân · cát đằng · **công bình** · công dã tràng · **công giáo** · **công nguyên** · -**cơ đốc giáo** · cường thịnh · cầm đuốc chơi đêm · cầu lam · cầu lộc · cầu ô · cẩm châu · -cẩm đường · cửu kinh · **cựu ước** · diêm vương · diêm vương tinh · dã hạc · dương quan · -dương đài · **dường như** · **giao tử** · giấc mai · giọt châu · giọt tương · hoa cái · -**hoa kiều** · **hoà đồng** · hoá công · hoả tinh · hán học · hán tộc · hán tự · hán văn · -hạ thần · hải phòng · hải triều · hải vương tinh · hằng nga · **hết sảy** · **hệ mặt -trời** · hội gió mây · hội long vân · **hồi giáo** · **khổng giáo** · kim tinh · **kinh -thánh** · la ve · lâm viên · **lưỡi hái** · lạc hầu · mạnh thường quân · **mặt trời** · nam -lâu · **nguyên đán** · **nho học** · nhân kiệt · năm cha ba mẹ · nước dương · nếm mật nằm -gai · nợ như chúa chổm · phong thu · **phù phiếm** · **phật giáo** · phật học · phật pháp · -phật tiền · phật tổ · phật tự · **phật đản** · quang phục · quyết tiến · quân thiều · **quả -đất** · **sao hôm** · sao hỏa · sao kim · **sao mai** · sao mộc · song mai · sơn cương · -sơn lâm · sơn mai · sơn nguyên · sơn trung · sư tử hà đông · **sừng trâu** · tam dân · tam -phủ · tam sơn · thanh nghị · thanh phong · thanh quang · thanh vận · thiên chúa · **thiên -chúa giáo** · thiên vương tinh · thuỷ liễu · thân giáp · thạch bàn · thạch thán · **thần -chết** · **tin lành** · tiên sư · **trái đất** · trướng huỳnh · trường giang · trường xuân · -tu lý · tuần duyên · tân kỳ · **tân ước** · tây dương · tây thiên · tô hạp · tùng lâm · tế -tân · **tết nguyên đán** · **tổ quốc** · tử phòng · u minh · vinh thăng · **việt ngữ** · -vách quế · vân hà · vân hán · vân trình · võ miếu · **văn giáo** · **văn miếu** · **văn -nhân** · văn quan · **văn võ** · văn đức · **vũ công** · **vũ hội** · vũng tàu · vương bá · -vạn an · vạn kiếp · vạn phúc · vầng ô · **vớ bở** · **xe tơ** · xuân hoá · xuân phong · -xuân tình · xích thố · xương thịnh · yên bình · yên chi · yên hoa · yên hà · đinh điền · -**đường luật** · **đường thi** · đại danh · đỉnh giáp non thần · **địa cầu** · đỗ vũ - -Bold: everyday or dictionary-headword vocabulary by this auditor's reading, ~55 of 215. -The rest are Sino-Vietnamese literary terms, religious/astronomical names that Wiktionary -capitalizes by convention, and genuine place names (`hải phòng`, `vũng tàu`, `bồng sơn`). - -**Narrowing does not fix this.** The plan's pre-decided narrowing — drop only when every -syllable is capitalized — was measured: drops fall 4,747 → 4,337, casualties fall only -215 → 147 (`mặt trời`, `trái đất`, `tổ quốc`-style two-cap forms stay dropped), and 410 -proper nouns come back in, including junk like `bản mẫu:-vie-n-` and `con kde`. Rejected. - -## 3. Corpus diff — old shipped vs new - -| | | -|---|---| -| overlap | 21,320 | -| gained | 1,099 (`gân bò`, `rèn đúc`, `ích kỷ`, `đáng lý`, `thở hồng hộc`) | -| lost | **26,896** | -| — dropped as proper noun | 4,245 | -| — absent from the wiktionary branch | 22,651 | -| — other | 0 | - -## 4. Playability - -| metric | old | new | -|---|---|---| -| words | 48,216 | 22,419 | -| syllables | 6,676 | 5,485 | -| syllables that open a word | 5,049 | 4,052 | -| …with ≥2 continuations | 3,682 | 2,691 | -| dead-end syllables | 1,627 | 1,433 | - -Bot-versus-bot, `internal/bot` real-corpus tests, 60 games each: - -| | old | new | -|---|---|---| -| hard-vs-easy win rate (moves) | 98% (3.3) | 98% (3.5) | -| medium-vs-easy | 87% | 88% | -| hard-vs-medium (moves) | 72% (3.5) | 55% (2.9) | -| easy-vs-easy game length | 17.5 moves | **11.3 moves** | -| hard decision p95 | 5.2 ms | 1.1 ms | - -Both tests pass on both databases. Easy-vs-easy games are a third shorter: the thinner -graph runs out of continuations sooner, which is the depth loss the plan predicted. -Hard-vs-medium falling to 55% means the graph gives the stronger bot fewer ways to trap. - -## 5. Decision - -Drop rule on the sample: **pass** (1 common word in 100). - -Case casualties: **open — needs the owner's call**, because the options trade the -"wiktionary data only" decision against ~55 everyday words: - -- **Accept** the 215 losses as measured. -- **Exception list**: a checked-in list of words to keep despite capitalization, curated - from the 215 above. Cause-aligned, small, but a hand-maintained artifact. -- **Read hongocduc for case evidence only**: recovers all 215 mechanically, but uses GPL - data as an input even though none of its words ship. Must be stated in `ATTRIBUTION.md`. - -**Recorded decision (owner, 2026-09-08): abandon the capitalization rule.** Every -wiktionary-tagged word is kept regardless of case; proper nouns stay in the corpus as they -are in the shipped database today. The rule, its reject reason and `--report-drops` were -removed from the builder. - -## 6. Final corpus, without the rule - -``` -accepted 26845 distinct words (sources: wiktionary) -``` - -| metric | old | new | -|---|---|---| -| words | 48,216 | 26,845 | -| syllables | 6,676 | 5,709 | -| syllables that open a word | 5,049 | 4,158 | -| …with ≥2 continuations | 3,682 | 2,787 | -| dead-end syllables | 1,627 | 1,551 | -| overlap / gained / lost | | 25,565 / 1,280 / 22,651 | - -Every lost word is absent from the 2018 wiktionary branch; none is lost to a rule. - -Bot-versus-bot, 60 games each: hard-vs-easy 95% (2.7 moves), medium-vs-easy 87%, -hard-vs-medium 63% (3.2 moves), easy-vs-easy **12.9 moves** (old: 17.5). Both real-corpus -tests pass. diff --git a/plans/260908-1525-dictionary-corpus-switch/phase-01-start.md b/plans/260908-1525-dictionary-corpus-switch/phase-01-start.md deleted file mode 100644 index a62fd56..0000000 --- a/plans/260908-1525-dictionary-corpus-switch/phase-01-start.md +++ /dev/null @@ -1,118 +0,0 @@ ---- -phase: 1 -title: "Phase 1: Read the merged list" -status: done -priority: P1 -effort: "3h" -dependencies: [] ---- - -# Phase 1: Read the merged list - -## Overview - -Teach `build-dictionary` a third input: undertheseanlp's merged JSONL. It selects words by -source membership and hands them to the normalization and filtering path that already -exists. - -## Requirements - -- Functional: read `{"text": "...", "source": ["hongocduc", ...]}` lines; keep a word when - its sources intersect the allowed set — **`wiktionary` only** by default. -- Functional: `hongocduc` and `tudientv` rows are skipped before any decision is made about - a word. They inform nothing. -- Functional: capitalization is not a filter. `Hà Nội` and `hà nội` are the same word and - land as the lowercase form, exactly as every other input mode already behaves. -- Functional: the `--min-words` default moves from 40,000 to 20,000. The new corpus is - ~26,800 words; the old floor would reject it. -- Non-functional: streaming line-by-line. The file is 4.8 MB; nothing here needs to be - clever. -- Non-functional: everything downstream — `accept()`, syllable indexing, alias generation, - `verify()`, `meta` — is reused unchanged. - -## Architecture - -``` ---merged --sources wiktionary - │ - ▼ one row at a time - sources ∩ allowed ≠ ∅ ? ── no ──▶ skipped, never counted - │ yes - ▼ - accept() (NFC, lowercase, ≥2 syllables, alphabet, phonotactics) - │ - ▼ - finish() (floor, aliases, atomic write, verify) ← shared with --words and --in -``` - -The `--sources` flag is kept general — a comma list validated against the three known -names — even though only `wiktionary` is passed, so a comparison build against another -branch is a flag away and never a code patch. Unknown names are an error, not a no-op. - -`finish()` is new: the three input modes used to each carry their own copy of the floor -check, alias generation, write and verify. One copy is what keeps a fixture from drifting -into a different shape from the database production loads. - -## Related Code Files - -- Created: `server/cmd/build-dictionary/merged_list.go` -- Created: `server/cmd/build-dictionary/merged_list_test.go` -- Modified: `server/cmd/build-dictionary/main.go` — `--merged` and `--sources` flags, - mutually-exclusive input check, `finish()`, `sourceSpec.url` / `sourceSpec.extra`, meta - rows, package doc comment, `--min-words` default -- Unchanged: `server/cmd/build-dictionary/filter.go` - -## Implementation Steps - -1. Write `merged_list.go`: decode with `encoding/json` per line (`bufio.Scanner` with a - raised buffer — a scanner that silently truncates is a bad way to lose words), skip rows - with no allowed source, feed the rest to `accept()`. -2. Add `--merged ` and `--sources wiktionary` (the default) to `main.go`. Change the - `--min-words` default to 20,000 in the same edit; the fixture path passes its own floor - explicitly and is unaffected. -3. Dispatch in `run()`: exactly one of `--merged`, `--words`, `--in` may be given. Refuse - two rather than picking one. -4. Write `source_url`, `source_commit`, `sources_kept` and `sources_excluded` into `meta` — - provenance in the artifact, not only in a markdown file. -5. Tests in `merged_list_test.go`, table-driven, over hand-written JSONL fixtures: - - `Hà Nội` tagged wiktionary → kept as `hà nội` - - a `hongocduc`-only word and a `tudientv`-only word → absent from the output - - `học sinh` in all three, plus `Học sinh` → kept once - - a malformed line → a clear error naming the line number, not a skipped word - - `--sources hongocduc,wiktionary` → strictly more words; `tudientv` still absent - - meta rows as above -6. Run against the real file and record the counts. - -## Result - -``` -rejected 2: contains a digit -rejected 147: contains punctuation -rejected 5335: fewer than 2 syllables -rejected 87: no Vietnamese letters -accepted 26845 distinct words (sources: wiktionary) -generated 1507 spelling aliases -``` - -The first cut of this phase also implemented a capitalization-based proper-noun drop -(`only ever capitalized` → rejected, `--report-drops` for audit). It reached the Phase 3 -gate, was audited, and was removed by the owner's decision — see `plan.md` and -`audit-proper-noun-drops.md`. Nothing of it remains in the code. - -## Success Criteria - -- [x] `go test ./cmd/build-dictionary/` passes, new tests included. -- [x] Building from the real merged file accepts >20,000 words (26,845). -- [x] Ordinary-word probes present: `học sinh`, `bánh mì`, `xe đạp`, `giáo viên`, `hoa - hồng`, `tình yêu`, `nước mắm`, `mặt trời`, `trái đất`. `con người` and `cánh diều` are - not in the 2018 branch at all — expected, recorded. -- [x] `--sources wiktionary` (default) and `--sources hongocduc,wiktionary` both build, with - the first strictly smaller (26,845 vs 61,271 with no case rule). -- [x] No word in the output is reachable only through `hongocduc` or `tudientv`. - -## Risk Assessment - -**A JSON schema surprise.** The file is 8 years old and unversioned; a row could carry an -unexpected shape. Signal: decode errors on the real file. Response: the error names the -line — inspect it, and only then decide between a tolerant skip with a counted reason and -a hard failure. Silent skipping is not an option. (Observed: none; all 79,226 rows decode.) diff --git a/plans/260908-1525-dictionary-corpus-switch/phase-02-pin-the-source.md b/plans/260908-1525-dictionary-corpus-switch/phase-02-pin-the-source.md deleted file mode 100644 index 9a79bea..0000000 --- a/plans/260908-1525-dictionary-corpus-switch/phase-02-pin-the-source.md +++ /dev/null @@ -1,87 +0,0 @@ ---- -phase: 2 -title: "Phase 2: Pin the source" -status: done -priority: P1 -effort: "2h" -dependencies: [1] ---- - -# Phase 2: Pin the source - -## Overview - -Point the build machinery at the new file: pinned by commit and checksum, fetched by -`make fetch-dict` and by the Docker dictionary stage, with the existing agreement test -still guarding both. - -## Requirements - -- Functional: `make fetch-dict` downloads the pinned JSONL and verifies its SHA-256; - `make dict` builds `data/noitu.db` from it. -- Functional: the Docker `dict` stage does the same, and its fixture escape hatch - (`FIXTURE_DICT=1`) keeps working untouched. -- Functional: the fetched source file is gitignored — we distribute the derived database, - never a copy of their file. -- Non-functional: `DICT_URL` / `DICT_SHA256` keep their names so - `web/tests/dictionary-source.test.js` continues to guard Makefile/Dockerfile agreement - with no change to the test. - -## Architecture - -The pin is a commit-addressed raw URL, which is immutable — unlike `master`, where the -URL and the checksum could silently diverge: - -```make -DICT_URL := https://raw.githubusercontent.com/undertheseanlp/dictionary/2c078cfc373b06e2980d324ce1d7bd13740c3319/dictionary/words.txt -DICT_SHA256 := 4c3e0e6117e4bdfa97731e135c3d4a05881889909267394a8de8d88ef79f13f0 -DICT_SRC := data/undertheseanlp-words.jsonl -``` - -`fetch-dict` and `verify-dict` keep their current shape — `curl` then `sha256sum -c` — -because that is already the right shape; only the size in the help text changes. - -## Related Code Files - -- Modify: `Makefile` — `DICT_URL`, `DICT_SHA256`, `DICT_SRC`, the `dict` target's flags - (`--merged` in place of `--in`), and the `help` text's "~179 MB" -- Modify: `Dockerfile` — `ARG DICT_URL`, `ARG DICT_SHA256`, the `curl`/`sha256sum` lines, - the `build-dictionary` invocation, and the `FIXTURE_DICT` comment's 179 MB reference -- Modify: `.gitignore` — ignore `data/undertheseanlp-words.jsonl` (`data/*.db` already - covers the derived and old source databases) -- Verify unchanged: `web/tests/dictionary-source.test.js`, `.github/workflows/ci.yml` - -## Implementation Steps - -1. Update the three Makefile variables and the `dict` target to pass `--merged $(DICT_SRC)`. - `--sources` is left at its `wiktionary` default so the Makefile and Dockerfile carry - one fewer thing to keep in agreement. -2. Reword `help`, `fetch-dict` and the `$(DICT_SRC)` guard: the download is ~4.8 MB now, - and saying "179 MB" would be the kind of stale comment that outlives three refactors. -3. Mirror all of it in the Dockerfile stage. Keep `FIXTURE_DICT=1` on `--words` — the - image smoke test must not start needing a network. -4. Add the source file to `.gitignore` and confirm `git status` is clean after a fetch. -5. Run `make fetch-dict && make verify-dict && make dict` end to end from a clean `data/`. -6. Run `npm test` in `web/` — `dictionary-source.test.js` should pass untouched. If it - needs editing, the variable names were changed unnecessarily; put them back. -7. Build the image with and without `FIXTURE_DICT=1` and confirm both produce a database. - -## Success Criteria - -- [ ] `make fetch-dict` pulls 4,813,111 bytes and the checksum verifies. -- [ ] `make dict` produces `data/noitu.db` with the Phase 1 word count. -- [ ] `web/tests/dictionary-source.test.js` passes with no edits to it. -- [ ] `docker build` succeeds both with `FIXTURE_DICT=1` and without. -- [ ] `git status` is clean after a fetch — no source file staged, ever. -- [ ] No file in the repository still claims a 179 MB dictionary download. - -## Risk Assessment - -**`raw.githubusercontent.com` is not an archive.** An 8-year-dormant repository could be -deleted or renamed, and then `make fetch-dict` fails for everyone. Signal: a 404 on fetch. -Response: the checksum makes any mirror verifiable, so host a copy as a release asset on -this repository and re-pin to it. Worth doing pre-emptively if the build ever gates CI. - -**The Docker stage and the Makefile drift.** Signal: the agreement test fails. Response: -that is the test doing its job — fix the file that is behind, and do not weaken the test to -match. diff --git a/plans/260908-1525-dictionary-corpus-switch/phase-03-audit-the-corpus.md b/plans/260908-1525-dictionary-corpus-switch/phase-03-audit-the-corpus.md deleted file mode 100644 index b27771b..0000000 --- a/plans/260908-1525-dictionary-corpus-switch/phase-03-audit-the-corpus.md +++ /dev/null @@ -1,110 +0,0 @@ ---- -phase: 3 -title: "Phase 3: Audit the corpus" -status: done -priority: P1 -effort: "2h" -dependencies: [1, 2] ---- - -# Phase 3: Audit the corpus - -> **Outcome.** Executed 2026-09-08; see [`audit-proper-noun-drops.md`](./audit-proper-noun-drops.md). -> The drop rule passed its sample gate (1 common word in 100) but cost 215 words under -> wiktionary-only case evidence, including everyday vocabulary. Narrowing was measured and -> rejected. **The owner abandoned the rule**: every word is kept regardless of case. Final -> corpus 26,845 words; playability and bot results recorded in the audit note. - -## Overview - -The gate. A rule that removes thousands of words on a capitalization heuristic has to be -looked at before it ships. This phase samples what was dropped, measures what the corpus -gained and lost, and confirms the game graph is not worse. - -## Requirements - -- Functional: a hand audit of 100 randomly sampled proper-noun drops, recorded in the - plan directory with the verdict per entry. -- Functional: a measured diff against the current `data/noitu.db` — gained, lost, and the - reason for each loss bucket. -- Functional: a playability comparison on the graph the game actually walks. -- Non-functional: every number in this phase is reproducible from a command written down - in the audit note, not from a one-off shell line nobody kept. - -## Architecture - -Three measurements, each against the freshly built database and the current one: - -1. **Drop audit.** Sample 100 of the dropped keys. Each is *proper noun* (correct), - *common word* (a false positive, the thing we are looking for), or *unclear*. The rule - passes at ≤2 common words in 100; that tolerance is the difference between a filter and - a corpus edit. -2. **Corpus diff.** Overlap, gained, lost. Split the lost into: dropped as proper nouns, - absent from the wiktionary branch. The report measured 4,335 / 22,651 of 26,986 lost; - this re-measures from the shipped build. Separately count the **271 case casualties** - — drops that have a lowercase form in `hongocduc` or `tudientv` — and list them in - full; that list is what the pass/narrow/exception decision is made on. -3. **Playability.** Words, syllables, syllables that can open a word, how many have ≥2 - continuations, and dead-end syllables — the last of these being what decides whether - the game hands somebody an unanswerable position. Current: 48,216 / 6,676 / 5,049 / - 3,682 / 1,627. Expected: 22,310 / 5,484 / 4,050 / 2,686 / 1,434. The corpus is smaller - by design, so the question is not "is it bigger" but "does the game still run": add - bot-versus-bot game lengths from `server/internal/bot`'s real-corpus tests on both - databases. - -## Related Code Files - -- Create: `plans/260908-1525-dictionary-corpus-switch/audit-proper-noun-drops.md` — - the sample, the verdicts, the commands that produced them -- Read only: `data/noitu.db` (current, before regeneration), the newly built database - -## Implementation Steps - -1. Build the new database to a scratch path, leaving the shipped one in place for - comparison. Do not overwrite `data/noitu.db` in this phase. -2. Emit the dropped keys from the builder — a `--report-drops ` flag, or a temporary - local patch if the flag would otherwise have no user. Prefer the flag: the next person - to change this rule will want it too. -3. Sample 100 with a fixed seed, write them into the audit note, and judge each one by - hand. Vietnamese place and person names are the expected content; anything that reads - as an ordinary word is a false positive and gets called one. -4. Run the corpus diff and the playability comparison; put both tables in the audit note. -5. Build `--sources hongocduc,wiktionary` to a second scratch path purely to enumerate the - case casualties (words dropped under `wiktionary` alone but kept when hongocduc's - lowercase forms are visible). Nothing from that build ships. -6. Decide: pass, narrow, exception list, or abandon the rule. Record the decision and its - reason in the audit note — and if the rule changes, Phase 1's tests change with it. - -## Success Criteria - -- [ ] The audit note exists, with 100 judged entries and the commands that produced them. -- [ ] ≤2 of 100 sampled drops are ordinary words, the 271 known case casualties aside — - those are listed in full and judged as a group. -- [ ] The new corpus has >20,000 words. -- [ ] The five graph numbers and bot-game lengths are recorded for both databases; bot - games on the new graph complete without the engine running out of moves earlier - than on today's. -- [ ] The loss buckets are quantified, not estimated. -- [ ] A pass/narrow/exception-list/abandon decision is written down with its reason. - -## Risk Assessment - -**The audit passes on a sample and the corpus is still wrong in the tail.** 100 of several -thousand is a sample, not a proof. Signal: players disputing rejected words after release. -Response: the drop list is a file — a disputed word can be checked against it in seconds, -and a per-word exception list is a small change on top of this design. - -**The audit fails and the phase becomes a redesign.** Signal: >2 common words in 100. -Response: apply Phase 1's pre-decided narrowing (every syllable capitalized) or the -exception list, re-audit once. If it fails again, ship without the drop — the proper-noun -problem then stays exactly as bad as it is today rather than getting worse. - -**The corpus is too thin in play.** 22,310 words is less than half of today's. Signal: bot -games noticeably shorter, or the dead-end rate per move up rather than down. Response: this -is the trigger for the documented upgrade path — the 2026-09-01 viwiktionary dump (31,637 -words, same license) as a second `--words` source — not for re-admitting GPL data. - -**Playability regresses in a way these five numbers do not capture.** Signal: aggregate -counts look fine but bot games end oddly short or long. Response: `server/internal/bot`'s -real-corpus tests run bot-versus-bot games on the shipped database; run them before -accepting the phase, since they exercise the graph rather than counting it. diff --git a/plans/260908-1525-dictionary-corpus-switch/phase-04-update-the-attribution.md b/plans/260908-1525-dictionary-corpus-switch/phase-04-update-the-attribution.md deleted file mode 100644 index 2295ca6..0000000 --- a/plans/260908-1525-dictionary-corpus-switch/phase-04-update-the-attribution.md +++ /dev/null @@ -1,116 +0,0 @@ ---- -phase: 4 -title: "Phase 4: Update the attribution" -status: done -priority: P1 -effort: "1h" -dependencies: [3] ---- - -# Phase 4: Update the attribution - -> **Outcome.** Done 2026-09-08. `data/LICENSE` untouched; `NOTICE` section 2 names the new -> upstream, `data/ATTRIBUTION.md` rewritten (eight modifications, both attribution links, -> both exclusions). Also updated, beyond the plan's list: the frontend attribution footer -> (`AttributionFooter.svelte`, `i18n/vi.js`) and its e2e assertion now credit Wiktionary -> tiếng Việt instead of minhqnd — the user-visible half of the CC BY-SA obligation. - -## Overview - -The data license does not change — the wiktionary branch is CC BY-SA 4.0, the same license -`noitu.db` already carries — but everything that names the source does. This phase rewrites -the attribution and notice text and regenerates the shipped database in one commit, so the -attribution never describes a database other than the one in the tree. - -## Requirements - -- Functional: `data/LICENSE` is untouched — still the CC BY-SA 4.0 text. -- Functional: `NOTICE` section 2 names the new upstream, keeps CC BY-SA 4.0, and keeps - section 1 (Apache-2.0 for code) byte-identical. -- Functional: `data/ATTRIBUTION.md` records the new source, the branch actually used, the - two branches excluded and why, and the full list of modifications. -- Functional: the regenerated `data/noitu.db` carries matching `meta` provenance. -- Non-functional: no overclaiming. The branch is a 2018 scrape of vi.wiktionary.org made by - a third party; say that, and attribute Wiktionary's contributors as CC BY-SA requires. - -## Architecture - -The dual-license structure in `NOTICE` is correct and stays; only the source lines of its -second section change: - -``` -1. SOURCE CODE — Apache-2.0 (unchanged: server/, tools/, web/, proto/) -2. DICTIONARY DATA — CC BY-SA 4.0 (unchanged license, new upstream) - data/noitu.db, data/LICENSE, data/ATTRIBUTION.md -``` - -The attribution chain is now two links instead of an opaque aggregate: vi.wiktionary.org -contributors (CC BY-SA) → `undertheseanlp/dictionary`, branch data `wiktionary`, commit -`2c078cf` → this project. Both links are named. - -The modifications list stays at eight: source selection replaces the old language filter -as entry 1, which the license requires us to declare as a change. - -## Related Code Files - -- Modify: `NOTICE` — section 2's upstream line, affected-artifacts line (`dictionary.db` - is no longer the source), and the closing paragraph about build artifacts -- Modify: `data/ATTRIBUTION.md` — source table, branch selection, exclusions, modifications - 9 and 10, the reproduce block -- Modify: `README.md` — dictionary-source lines (`~179 MB`, `dictionary.db`, `minhqnd`, - the manual `curl` + `--in` example) -- Modify: `docs/deployment.md` — the 179 MB builder-stage sentence -- Verify unchanged: `data/LICENSE`, `.github/workflows/ci.yml` required-files check -- Regenerate: `data/noitu.db` - -## Implementation Steps - -1. Rewrite `NOTICE` section 2's source lines: upstream `undertheseanlp/dictionary` at - commit `2c078cf`, wiktionary data only; affected artifact `data/noitu.db`. Leave the - share-alike paragraph as is — it is still true. -2. Rewrite `data/ATTRIBUTION.md`: - - source is `undertheseanlp/dictionary`, file `dictionary/words.txt`, commit `2c078cf` - - rows used: those tagged `wiktionary` — a 2018-12-10 scrape of vi.wiktionary.org, - whose text is CC BY-SA 4.0 / GFDL by its contributors - - `hongocduc` **excluded**: GNU GPL per the branch README; using it would relicense this - database to GPLv3 - - `tudientv` **excluded**: its README declares copyright *"Chưa rõ"* and names - Soha/Vietlex (Hoàng Phê) as its primary source. Say it plainly so nobody reopens it. - - modifications renumbered: **1. source selection** replaces the language filter (the new - file is Vietnamese-only); the other seven carry over, with normalization stating that - capitalization removes nothing — the proper-noun drop was abandoned at the Phase 3 gate - - reproduce block: `make fetch-dict` is ~4.8 MB now -3. Grep the repository for `minhqnd`, `dictionary.db`, `179` and `--in` and fix every - survivor outside `plans/` — the reports are dated records and stay as written. -4. Regenerate `data/noitu.db` and confirm its `meta` rows agree with `ATTRIBUTION.md`: - source URL, commit, license, `sources_kept = wiktionary`, - `sources_excluded = hongocduc,tudientv`. -5. Run the CI required-files check locally, or read it and confirm by hand. - -## Success Criteria - -- [ ] `data/LICENSE` has no diff. -- [ ] `NOTICE` section 2 names the new upstream and still says CC BY-SA 4.0; section 1 is - byte-identical. -- [ ] `data/ATTRIBUTION.md` lists eight modifications, names the branch used and the two - excluded with reasons, and credits vi.wiktionary.org contributors. -- [ ] `meta` in the built database matches the attribution file. -- [ ] No file outside `plans/` names minhqnd, `dictionary.db` as a source, or a 179 MB - download. -- [ ] CI's required-files check passes. - -## Risk Assessment - -**A stale source claim survives somewhere.** A README line or Docker comment naming the old -upstream is an attribution misstatement, not a typo. Signal: the grep in step 3 finds -something after the phase is called done. Response: grep is a step in the phase for that -reason; run it last, not first. - -**Section 1 gets damaged while section 2 is edited.** The code's Apache-2.0 grant is not -in scope and must come out byte-identical. Signal: a diff touching section 1. Response: -review the `NOTICE` diff before committing. - -**The attribution credits the wrong party.** CC BY-SA attribution belongs to Wiktionary's -contributors; undertheseanlp is the intermediary that scraped and redistributed. Signal: -an `ATTRIBUTION.md` that names only the GitHub repository. Response: step 2 names both -links of the chain explicitly. diff --git a/plans/260908-1525-dictionary-corpus-switch/phase-05-retire-the-sqlite-path.md b/plans/260908-1525-dictionary-corpus-switch/phase-05-retire-the-sqlite-path.md deleted file mode 100644 index d4fdd9f..0000000 --- a/plans/260908-1525-dictionary-corpus-switch/phase-05-retire-the-sqlite-path.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -phase: 5 -title: "Phase 5: Retire the SQLite path" -status: done -priority: P2 -effort: "1h" -dependencies: [4] ---- - -# Phase 5: Retire the SQLite path - -> **Outcome.** Done 2026-09-08. `--in`, `--table`, `--word-col`, `--lang-col`, `--lang`, -> `resolveSource`, `validateSource`, `listTables`, `listColumns`, `pickColumn`, `extract` -> and `quoteIdent` removed; `meta` no longer writes `source_word_column` / -> `source_lang_column` (no consumer existed). The four auto-detection tests went; the -> output-shape tests now build their fixture through `--merged`. `--help` lists `--merged`, -> `--sources`, `--words`, `--out`, `--max-syllables`, `--min-words` and nothing else. - -## Overview - -Delete the input mode nothing uses any more: reading a source SQLite database, with the -table and column auto-detection that existed because the upstream schema was not ours to -rely on. Two input modes for one source is dead weight, and dead weight in a build tool is -where stale assumptions hide. - -## Requirements - -- Functional: `--in`, `--table`, `--word-col`, `--lang-col` and `--lang` are gone, along - with the schema resolution behind them. -- Functional: `--merged` (the corpus) and `--words` (the fixture) remain. The fixture path - stays because the e2e suite and the Docker smoke build depend on it. -- Non-functional: the package doc comment describes what the tool now does, not what it - used to. -- Non-functional: `modernc.org/sqlite` stays a dependency — it writes the output database. - -## Architecture - -What `run()` dispatches on collapses from three modes to two: - -``` ---merged the corpus: undertheseanlp JSONL (default path) ---words the fixture: one word per line (e2e, Docker smoke build) -``` - -`resolveSource` and its auto-detection tests go with `--in`. They were never about the -game's rules — they were insurance against an upstream schema we did not control, and we -no longer read that upstream. - -## Related Code Files - -- Modify: `server/cmd/build-dictionary/main.go` — remove `--in` and the four schema flags, - `resolveSource`, the SQLite read path, and the `config` fields behind them; rewrite the - package doc comment and the usage example -- Modify: `server/cmd/build-dictionary/main_test.go` — remove the source-resolution tests; - keep and extend the ones about output shape -- Modify: `Makefile` — drop any remaining `--in` reference and 179 MB wording from `help` -- Verify: `Dockerfile` no longer mentions `--in` - -## Implementation Steps - -1. Remove the flags, the `config` fields, `resolveSource` and the SQLite read path. -2. Delete the tests that exist only to cover auto-detection. Do not delete tests covering - output shape, `verify()`, or the reject-reason counts — those still describe behavior. -3. Rewrite the package doc comment: the input is a 4.8 MB JSONL wordlist with source - membership and capitalization, of which only wiktionary-tagged rows are read; the - output is the game's syllable-indexed database. -4. `go build ./... && go vet ./... && go test ./...` from `server/`. -5. Grep for `--in`, `dictionary.db`, `resolveSource` and `lang_code` outside `plans/`; - nothing should survive except in the dated reports. -6. `make dict` once more from a clean `data/` to prove the surviving path is the one that - works. - -## Success Criteria - -- [ ] `go build`, `go vet` and `go test ./...` are clean from `server/`. -- [ ] `build-dictionary --help` lists `--merged` and `--words` and nothing about tables, - columns or language codes. -- [ ] `make dict` and `make fixture-dict` both work. -- [ ] The e2e suite still builds its fixture and passes. -- [ ] No reference to `--in`, `resolveSource` or `lang_code` survives outside `plans/`. - -## Risk Assessment - -**Deleting the wrong tests.** The auto-detection tests and the output-shape tests live in -the same file. Signal: coverage of `verify()` or the reject counters disappears with the -deletion. Response: read `main_test.go` in full before cutting, and delete by test -function rather than by line range. - -**Somebody wants the SQLite path back.** A future source could ship SQLite again. Signal: -a future plan proposing it. Response: git remembers, and re-adding a reader for a schema -we would then actually control is a smaller job than maintaining auto-detection for a -schema nobody reads. Deleting it is the cheaper bet. - -**The fixture path is quietly coupled to something removed.** Signal: `make fixture-dict` -or the e2e build fails after the cut. Response: step 6 builds both paths; run it before -calling the phase done, not after. diff --git a/plans/260908-1525-dictionary-corpus-switch/plan.md b/plans/260908-1525-dictionary-corpus-switch/plan.md deleted file mode 100644 index f234306..0000000 --- a/plans/260908-1525-dictionary-corpus-switch/plan.md +++ /dev/null @@ -1,155 +0,0 @@ ---- -title: "Dictionary corpus switch" -description: "Replace the 179 MB minhqnd aggregate with the wiktionary branch of undertheseanlp/dictionary only (hongocduc and tudientv excluded) and keep data/noitu.db on CC BY-SA 4.0" -status: completed -priority: P1 -effort: "~1d" -tags: [dictionary, data, licensing, build] -created: 2026-09-08 -updated: 2026-09-08 -blockedBy: [] -blocks: [] ---- - -# Dictionary corpus switch - -## Overview - -`data/noitu.db` was derived from `minhqnd/dictionary` v2.0.0: a 179 MB SQLite aggregate of -five upstreams, redistributed as CC BY-SA 4.0. Its license chain does not close — three of -its five inputs are GPL (FVDP, tudientv, "Vietnamese Explanatory Dictionary") and GPL does -not permit relicensing to CC BY-SA. - -This plan switches the corpus to the **`wiktionary` source inside `undertheseanlp/dictionary`** -— a single 4.8 MB JSONL file carrying, per word, which of three source dictionaries contain -it. Only rows whose `source` list contains `wiktionary` are read. `hongocduc` (GPL) is -excluded so the data license does not change; `tudientv` is excluded because its own README -declares copyright *"Chưa rõ"* and names Soha/Vietlex (Hoàng Phê) as its primary source. -`data/noitu.db` **stays CC BY-SA 4.0**: the wiktionary branch is a 2018-12-10 scrape of -vi.wiktionary.org, whose text is CC BY-SA. - -Evidence, all measured through the real `build-dictionary` filter: -`plans/reports/research-260908-1507-undertheseanlp-dictionary.md` (branch licensing), -`plans/reports/research-260908-1529-viwiktionary-dump-measured.md` (yields, alternatives) -and [`audit-proper-noun-drops.md`](./audit-proper-noun-drops.md) (the Phase 3 gate). - -## What changes, in numbers - -| | before | after | -|---|---|---| -| words | 48,216 | **26,845** | -| syllables | 6,676 | 5,709 | -| syllables that can open a word | 5,049 | 4,158 | -| …with ≥2 continuations | 3,682 | 2,787 | -| dead-end syllables | 1,627 | 1,551 | -| source download | 179 MB SQLite | 4.8 MB JSONL | -| data license | CC BY-SA 4.0 | **CC BY-SA 4.0 (unchanged)** | -| `--min-words` floor | 40,000 | **20,000** | - -**This is a deliberate 44% cut.** 1,280 words gained, 22,651 lost — every one of them -simply absent from a 2018 wiktionary scrape (`bánh đúc`, `lò vi sóng`, `dân làng`, `thót -tim`). Bot-versus-bot games run about a quarter shorter on the thinner graph (easy-vs-easy -12.9 moves vs 17.5). This loss is accepted here; the upgrade path is written down below. - -## Decisions taken - -- **Only `wiktionary` rows are read.** `hongocduc` and `tudientv` inform nothing. That is - what keeps the license CC BY-SA 4.0 and what makes "we do not touch that data" true - without qualification. -- **Capitalization is not a filter.** The original design dropped words that appear only - ever capitalized, as proper nouns. It was built, run and audited at the Phase 3 gate: the - rule was precise on a 100-word sample (1 common word) but, with wiktionary-only case - evidence, it also removed 215 words including `mặt trời`, `trái đất`, `tổ quốc`, `dường - như`, `phật giáo`. Narrowing was measured and did not help. **The owner chose to keep - every word regardless of case.** Proper nouns like `hà nội` remain playable, exactly as - they are today. The mechanism was removed from the builder rather than left behind a flag. -- **Rejected alternatives, so nobody reopens them without new evidence:** - - `hongocduc + wiktionary` (61,271 words) — forces `noitu.db` to GPLv3. Rejected to keep - CC BY-SA 4.0. - - the 2026-09-01 viwiktionary dump (36,200 words without a case rule, same license, - monthly pin) — the 2018 branch is a near-subset of it. Rejected for now in favour of the - simpler single-file pin; **this is the documented upgrade path** if the 27k corpus - proves too thin in play. See the 15:29 report. - - kaikki.org enwiktionary Vietnamese — unpinnable rolling file, deprecated by its host. -- **The SQLite input path is deleted, not kept "just in case."** The fixture path - (`--words`) stays because the e2e suite and the Docker smoke build use it. -- **The build floor drops to 20,000.** The old 40,000 default would reject the new corpus; - it moved once, in `main.go`. - -## The pin - -``` -URL https://raw.githubusercontent.com/undertheseanlp/dictionary/2c078cfc373b06e2980d324ce1d7bd13740c3319/dictionary/words.txt -SHA256 4c3e0e6117e4bdfa97731e135c3d4a05881889909267394a8de8d88ef79f13f0 -Size 4,813,111 bytes — 79,226 JSONL rows, 32,484 of them tagged wiktionary -Commit 2c078cf (2018-12-10, the repository's last) -``` - -Pinned by commit rather than branch: `raw.githubusercontent.com///` is -immutable, so the URL and the checksum cannot disagree later. The variable names -`DICT_URL` / `DICT_SHA256` were kept so `web/tests/dictionary-source.test.js` needed no -change — only their values moved. - -## Phases - -| # | Phase | Status | -|---|-------|--------| -| 1 | [Read the merged list](./phase-01-start.md) | Done | -| 2 | [Pin the source](./phase-02-pin-the-source.md) | Done | -| 3 | [Audit the corpus](./phase-03-audit-the-corpus.md) | Done — gate sent the case rule back; rule abandoned | -| 4 | [Update the attribution](./phase-04-update-the-attribution.md) | Done | -| 5 | [Retire the SQLite path](./phase-05-retire-the-sqlite-path.md) | Done | - -All five phases are uncommitted in one working tree; they should land together so the -attribution never describes a database other than the one the build produces. - -## Success criteria - -- [x] `make fetch-dict && make dict` builds `data/noitu.db` from a 4.8 MB pinned file with - no 179 MB download. -- [x] The shipped database has **more than 20,000 words** and **no word that reaches it - through `hongocduc` or `tudientv` alone**. -- [x] `meta` records the source URL, commit, sources kept and sources excluded. -- [x] Playability is measured and recorded: syllables, openers, continuations, dead ends, - and bot-versus-bot game lengths on the new graph versus the old. -- [x] `data/LICENSE` is unchanged; `NOTICE` and `data/ATTRIBUTION.md` name the new source, - its commit and license, and record why the two other branches are excluded. The - frontend attribution footer credits Wiktionary tiếng Việt. -- [x] `go vet ./... && go test ./...` (incl. `-race` on the touched packages), `npm run - check && npm test`, and the Docker image (`noitu:ci` with the CI file checks, plus - `FIXTURE_DICT=1`) are green; the fixture build path is untouched. -- [ ] The Playwright e2e suite is green. **Not verified locally**: Playwright's Chromium - download timed out twice on this machine (2026-09-08), so every test failed at browser - launch, not on game behaviour. The one e2e assertion this plan changes (the attribution - footer's link text and href in `web/e2e/bot-game.spec.js`) was updated by hand. Verify - in CI or after `npx playwright install chromium` succeeds. -- [x] `--in` and its schema auto-detection are gone, and no file outside `plans/` still - mentions minhqnd, `dictionary.db`, `--in` or a 179 MB download. - -## Licence version — decided after review - -The owner asked whether the data could move to Apache-2.0 to simplify the repo. It cannot: -the words are Wiktionary contributors' text under a share-alike licence, and neither -undertheseanlp nor this project can relicense them. The owner then chose to **keep the -licence the source text actually carried, CC BY-SA 3.0 Unported**, rather than upgrade the -derivative to 4.0 under the later-version clause. `data/LICENSE` is now the BY-SA 3.0 legal -code; `NOTICE`, `ATTRIBUTION.md`, README, Dockerfile, deployment doc, the builder's meta -string and the in-game footer all say 3.0. The "unchanged" wording above predates this. - -## Review - -Code review (2026-09-08) returned DONE_WITH_CONCERNS; every finding was applied: the -builder's copy of the upstream commit is now guarded by `web/tests/dictionary-source.test.js` -alongside the Makefile/Dockerfile pin; fixture builds record "no upstream data" instead of -inheriting the CC BY-SA string; `ATTRIBUTION.md` states the 2018 snapshot was CC BY-SA 3.0 -and why the derived database is 4.0; dead `contains()`, redundant sorts, the off-by-one -scanner line number, stale "48k" comments, the Makefile `curl` flags and the Dockerfile -download name were fixed; input-selection and fixture-licence tests were added. - -## Open questions - -- Is 26,845 words enough for the game to feel playable? Bot games are ~25% shorter; only - real play can say whether that is felt. If not, the 2026 viwiktionary dump is the next - step, not a return to GPL data. - - diff --git a/plans/260908-1653-kaikki-viwiktionary-corpus/measurement.md b/plans/260908-1653-kaikki-viwiktionary-corpus/measurement.md deleted file mode 100644 index 738c4a7..0000000 --- a/plans/260908-1653-kaikki-viwiktionary-corpus/measurement.md +++ /dev/null @@ -1,97 +0,0 @@ -# Measurement: first kaikki build vs the undertheseanlp corpus - -Date: 2026-09-08. Phase 3 of `plan.md`. The pre-switch `data/noitu.db` (26,845 words) was -copied aside before the first kaikki build and is the "old" side of every table. - -## The file measured - -``` -source_url https://kaikki.org/viwiktionary/Ti%E1%BA%BFng%20Vi%E1%BB%87t/kaikki.org-dictionary-Ti%E1%BA%BFngVi%E1%BB%87t.jsonl -source_sha256 51ddc2fbd73cd7e2a468ccff3c31200928e7200cc58622f528ba3eb84c7a49f0 -source_rows 44564 -source_fetched_at 2026-09-08T10:04:08Z (kaikki's export of 2026-09-06, dump 2026-09-01) -size 62,288,004 bytes; longest line 61,552 bytes -``` - -Build log: - -``` -rejected 4: contains a digit -rejected 136: contains punctuation -rejected 8348: fewer than 2 syllables -rejected 160: no Vietnamese letters -parts of speech: noun 16219, verb 9355, adj 6698, name 5180, unknown 4652, adv 1024, - phrase 392, proverb 322, character 198, intj 175, pron 100, conj 70, prep 48, num 45, - abbrev 38, particle 18, romanization 14, prefix 13, suffix 2, det 1 -accepted 34813 distinct words -generated 1664 spelling aliases (552 skipped as ambiguous or already real words) -``` - -## Commands - -```sh -cp data/noitu.db /tmp/old.db # before the switch -make fetch-dict && make dict # or the raw curl + go run lines in README -cd server && go test ./internal/bot/ -run RealCorpus -v -count=1 # once per database at data/noitu.db -# graph numbers and diff: sqlite, see the queries below -``` - -```sql -SELECT COUNT(*) FROM words; -SELECT COUNT(*) FROM syllables; -SELECT COUNT(DISTINCT first) FROM words; -SELECT COUNT(*) FROM (SELECT first FROM words GROUP BY first HAVING COUNT(*) >= 2); -SELECT COUNT(*) FROM syllables WHERE out_degree = 0; -``` - -## Playability - -| metric | old (undertheseanlp) | new (kaikki) | -|---|---|---| -| words | 26,845 | **34,813** | -| syllables | 5,709 | 6,081 | -| syllables that open a word | 4,158 | 4,487 | -| …with ≥2 continuations | 2,787 | 3,163 | -| dead-end syllables | 1,551 | 1,594 (27.2% → 26.2% of syllables) | - -Bot-versus-bot, 60 games each, same run: - -| | old | new | -|---|---|---| -| hard-vs-easy win rate (moves) | 98% (3.6) | 97% (3.2) | -| medium-vs-easy | 87% | 95% | -| hard-vs-medium (moves) | 57% (3.1) | 67% (3.1) | -| easy-vs-easy game length | 14.4 moves | 14.4 moves | -| hard decision p95 | 1.5 ms | 3.1 ms | - -Both real-corpus tests pass on both databases. Game length is unchanged; the stronger bots -win more often on the denser graph, which is the expected direction. - -## Corpus diff - -| | | -|---|---| -| shared | 25,392 | -| gained | **9,421** — `công nghiệp hóa`, `bóng râm`, `bội thu`, `đồng dao`, `bỏ ngỏ`, `đắt hàng`, `ấn độ giáo`, `chạy làng`, `béo nung núc`, `nhìn muốn rụng trứng` | -| lost | **1,453** — see below | - -### The 1,453 lost words - -Sample of 25 (seed 7): `trịnh tuệ`, `tuân khanh`, `qua quít`, `cúc pha`, `mành mành`, -`cây bài chặt`, `trảm phong`, `kéo cánh`, `ngọt lự`, `rưng rức`, `làu nhàu`, `trong lúc`, -`khí hư`, `trần ích tắc`, `nháo nhác`, `trần cảnh`, `óc ách`, `lườm lườm`, `hỗn luân`, -`trụi lủi`, `tục tác`, `lảu nhảu`, `phát sầu`, `hãng hàng không quốc gia việt nam`, `trói ké`. - -Two kinds. Roughly a fifth are historical person names (`trần ích tắc`, `trần cảnh`, `trịnh -tuệ`) whose pages have since been deleted or moved. The rest are real vocabulary, heavily -reduplicatives (`rưng rức`, `nháo nhác`, `óc ách`, `làu nhàu`, `lườm lườm`), that the 2018 -scrape had and the current wiktextract export does not. Most likely those pages still exist -but in a markup the `vi` extractor does not yet parse — the raw 2026 dump has 36,200 -words to kaikki's 34,813, a 1,387-word gap of the same order. **Not acted on:** the plan -does not merge sources, and the net is +7,968 words. If the reduplicatives matter, the -option is a second `--words` input reduced from the raw dump, as the previous plan's -research described. - -## Verdict - -Every number above today's; nothing regresses. Corpus accepted as built. diff --git a/plans/260908-1653-kaikki-viwiktionary-corpus/phase-01-start.md b/plans/260908-1653-kaikki-viwiktionary-corpus/phase-01-start.md deleted file mode 100644 index 983ea53..0000000 --- a/plans/260908-1653-kaikki-viwiktionary-corpus/phase-01-start.md +++ /dev/null @@ -1,118 +0,0 @@ ---- -phase: 1 -title: "Phase 1: Read the kaikki file" -status: completed -priority: P1 -effort: "3h" -dependencies: [] ---- - -# Phase 1: Read the kaikki file - -## Overview - -Replace the undertheseanlp reader in `build-dictionary` with one for kaikki's wiktextract -JSONL, hashing the input as it streams so the database records exactly which bytes it was -built from. - -## Requirements - -- Functional: `--kaikki ` reads one JSON object per line, keeps rows whose - `lang_code` is `vi`, and feeds `word` to the existing `accept()`. -- Functional: rows with any other `lang_code` are counted and rejected, not silently - skipped — the file is vi-only today, and a change in that would be worth seeing in the log. -- Functional: `pos` is tallied per value and printed with the reject counts. It never filters. -- Functional: the SHA-256 of the file and the row count are computed during the single - streaming pass and written to `meta` as `source_sha256` and `source_rows`, with - `source_fetched_at` taken from the file's modification time. -- Functional: `--merged` and `--sources` are removed along with `merged_list.go` and its - tests. Exactly one of `--kaikki` / `--words` must be given, same rule as today. -- Functional: the `--min-words` default moves from 20,000 to 30,000. -- Non-functional: a malformed line is an error naming the line, as today. Lines can be - long — a row carries every sense and translation — so the scanner buffer must allow - several megabytes; measure the longest line on the real file and set the cap above it - with headroom, or switch to `bufio.Reader.ReadBytes('\n')` which has no cap. -- Non-functional: `finish()`, `accept()`, aliases, `write`, `verify` unchanged. - -## Architecture - -``` ---kaikki - │ one line at a time, bytes also fed to sha256.New() - ▼ - json.Unmarshal → {word, lang_code, pos} - │ - ├─ lang_code != "vi" → rejects["not Vietnamese-language entry"]++ - ▼ - accept(word) (NFC, lowercase, ≥2 syllables, alphabet, phonotactics) - ▼ - finish() (floor 30,000, aliases, atomic write, verify) -``` - -Decode only the three fields via a struct; `encoding/json` ignores the rest, so the 62 MB -of senses and translations cost I/O but not memory. - -`sourceSpec.extra` already exists for mode-specific meta rows; the kaikki provenance uses it: - -``` -source_url the kaikki URL (constant kaikkiSourceURL) -source_sha256 hex of the streamed bytes -source_rows lines decoded -source_fetched_at file mtime, RFC 3339 UTC -source_license CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) -``` - -## Related Code Files - -- Create: `server/cmd/build-dictionary/kaikki_list.go` -- Create: `server/cmd/build-dictionary/kaikki_list_test.go` -- Delete: `server/cmd/build-dictionary/merged_list.go`, `merged_list_test.go` -- Modify: `server/cmd/build-dictionary/main.go` — flags, dispatch, doc comment, `--min-words` - default, `mergedProvenance` call site -- Modify: `server/cmd/build-dictionary/main_test.go` — `fixtureSource`/`defaultRows` emit - kaikki-shaped rows (`{"word": ..., "pos": ..., "lang_code": "vi"}`); the "excluded source" - row becomes a `lang_code: "en"` row - -## Implementation Steps - -1. Write `kaikki_list.go`: `kaikkiRow{Word, Pos, LangCode}`, `readKaikkiList(path, maxSyllables)` - returning words, rejects, a `pos` tally, and a `kaikkiProvenance` (sha256, rows, mtime). - Wrap the file in `io.TeeReader` into `sha256.New()` so hashing is free. -2. Add `rejectNotVietnamese rejectReason = "not a Vietnamese-language entry"` to `filter.go`. -3. In `main.go`: replace `merged`/`sources` with `kaikki` in `config`, flags and `run()`; - `--min-words` default 30,000; `runFromKaikkiList` logs rejects, the POS tally - (sorted, one line) and the accepted count, then calls `finish()` with the provenance spec. -4. Delete the merged reader and its tests. Rewrite `main_test.go` fixtures to kaikki rows. -5. Tests in `kaikki_list_test.go`: a `vi` row kept; an `en` row rejected and counted; - `Hà Nội` kept as `hà nội`; a word listed twice with different `pos` kept once; a malformed - line names its number; `meta` carries a 64-hex `source_sha256` equal to `sha256sum` of the - fixture file, the right `source_rows`, and no `source_commit`/`sources_*` keys. -6. Run against the real file (`scratchpad/kk-vi-edition.jsonl` from this session, or a fresh - download) and record the counts in phase 3. - -## Success Criteria - -- [x] `go test ./cmd/build-dictionary/` passes; `merged_list*.go` no longer exist. -- [x] Real file builds >30,000 words (measured 34,813) and logs a POS tally. -- [x] `build-dictionary --help` lists `--kaikki`, `--words`, `--out`, `--max-syllables`, - `--min-words` and nothing else. -- [x] `meta.source_sha256` of a build equals `sha256sum` of the input file. -- [x] The fixture path (`--words ... --min-words 150`) is byte-for-byte unaffected in - behaviour: same 205 words from `testdata/fixture-words.txt`. - -## Risk Assessment - -**A row exceeds the scanner buffer.** Signal: `bufio.Scanner: token too long` on the real -file. Response: pre-decided — use `bufio.Reader` with no line cap rather than guessing a -buffer size; measure the longest line once and note it in the phase result. - -**kaikki changes the row shape.** `word` and `lang_code` are wiktextract's stable core fields -and unlikely to move. Signal: zero accepted words → the 30,000 floor fails the build with a -clear message. Response: read the new shape, adjust the struct; the failure mode is loud by -design. - -**The file arrives partially.** `curl -f` catches HTTP errors but not a truncated body, and -a body cut exactly on a line boundary parses cleanly. Signal: a malformed final line (error -names it) or a word count under the 30,000 floor (13.8% headroom on 34,813). Response: the -fetch writes to a `.part` name and renames only on success (review finding, Phase 2), so an -interrupted download is never left for the next `make dict`; the floor covers the rest. diff --git a/plans/260908-1653-kaikki-viwiktionary-corpus/phase-02-fetch-the-newest-file.md b/plans/260908-1653-kaikki-viwiktionary-corpus/phase-02-fetch-the-newest-file.md deleted file mode 100644 index 9752ac8..0000000 --- a/plans/260908-1653-kaikki-viwiktionary-corpus/phase-02-fetch-the-newest-file.md +++ /dev/null @@ -1,105 +0,0 @@ ---- -phase: 2 -title: "Phase 2: Fetch the newest file" -status: completed -priority: P1 -effort: "2h" -dependencies: [1] ---- - -# Phase 2: Fetch the newest file - -## Overview - -Point the Makefile and the Docker dictionary stage at kaikki's rolling URL with no checksum, -keep the two build files in agreement, and make sure a failed download cannot pass as a -success anywhere. - -## Requirements - -- Functional: `make fetch-dict` downloads the kaikki file to `data/kaikki-viwiktionary-vi.jsonl` - with `curl -fL` and fails on any HTTP error; `make dict` builds from it with `--kaikki`. -- Functional: `verify-dict` and `DICT_SHA256` are removed from the Makefile and Dockerfile; - nothing in the repository claims a checksum for this file. -- Functional: the Docker `dict` stage downloads the same URL and runs the same command; - `FIXTURE_DICT=1` keeps working untouched. -- Functional: `web/tests/dictionary-source.test.js` asserts the Makefile and Dockerfile URLs - agree, that the URL is the kaikki viwiktionary `Tiếng Việt` file, and that the builder's - `kaikkiSourceURL` constant is the same URL. The three checksum assertions go. -- Functional: the CI leak guard greps for `kaikki.*\.jsonl$`; `.gitignore` ignores the new - file name. -- Non-functional: no resume flag (`-C -`) — a rolling file can change between attempts, and a - resumed download would splice two versions together. - -## Architecture - -```make -DICT_URL := https://kaikki.org/viwiktionary/Ti%E1%BA%BFng%20Vi%E1%BB%87t/kaikki.org-dictionary-Ti%E1%BA%BFngVi%E1%BB%87t.jsonl -DICT_SRC := data/kaikki-viwiktionary-vi.jsonl -DICT_OUT := data/noitu.db - -fetch-dict: - @mkdir -p data - curl -fL -o $(DICT_SRC).part $(DICT_URL) && mv $(DICT_SRC).part $(DICT_SRC) - -dict: $(DICT_SRC) - cd server && go run ./cmd/build-dictionary --kaikki ../$(DICT_SRC) --out ../$(DICT_OUT) -``` - -The Dockerfile mirrors it with `ARG DICT_URL` only. The URL must stay percent-encoded in -both files: Make and `sh` would otherwise split on the space in `Tiếng Việt`. - -Correctness now rests on three checks instead of a checksum: `curl -f` plus an atomic -`.part` rename (HTTP errors and interrupted downloads), the JSON decoder (shape, and -truncation that lands mid-line), the 30,000 floor (content, including a truncation that -lands on a line boundary). - -## Related Code Files - -- Modify: `Makefile` — variables, `help` text, `fetch-dict`, remove `verify-dict` and its - `.PHONY` entry, `dict` flags -- Modify: `Dockerfile` — `ARG DICT_URL`, remove `ARG DICT_SHA256` and the `sha256sum` line, - download file name, `--kaikki`; stage comment (no longer "pinned … checked by digest") -- Modify: `web/tests/dictionary-source.test.js` — drop checksum tests, keep URL agreement, - add the kaikki-path and builder-constant assertions -- Modify: `.github/workflows/ci.yml` — leak-guard pattern and comment -- Modify: `.gitignore` — `data/kaikki-viwiktionary-vi.jsonl` replaces the undertheseanlp entry -- Modify: `README.md` — Make targets table (`verify-dict` row gone), the manual `curl` and - `build-dictionary` lines - -## Implementation Steps - -1. Update the Makefile variables and targets; delete `verify-dict`. -2. Mirror in the Dockerfile; rewrite the stage comment to say the file is fetched fresh and - identified by the SHA-256 the builder records. -3. Rewrite the pin test as described. Run `npx vitest run tests/dictionary-source.test.js`. -4. Update the CI leak guard, `.gitignore` and the README lines. -5. Run `make fetch-dict && make dict` (or the raw commands on Windows) from a clean `data/`. -6. Build the image with and without `FIXTURE_DICT=1`; run the CI file checks against the - real one and confirm no `*.jsonl` is inside. -7. Simulate a bad download once: point `DICT_URL` at a 404 and confirm `curl -f` fails the - target; feed the builder a truncated copy of the file and confirm the build fails. - -## Success Criteria - -- [x] `make fetch-dict` fetches ~62 MB and `make dict` builds >30,000 words. -- [x] `grep -rn DICT_SHA256 --exclude-dir=plans .` finds nothing; `make verify-dict` is not a - target. -- [x] `web/tests/dictionary-source.test.js` passes and fails if either URL is edited alone. -- [x] Both image variants build; the real one passes the CI file checks; no `.jsonl` inside. -- [x] A 404 URL and a truncated file each fail the build with a readable message. - -## Risk Assessment - -**kaikki.org is a single volunteer-run host.** An outage breaks `make fetch-dict` and the -release image build until it returns. Signal: `curl: (22)` on fetch. Response: wait, or -build from a previously fetched local copy (`make dict` needs only the file); if outages -recur, keep a copy as a release asset and point `DICT_URL` at it — the open question in -`plan.md`. - -**Two builds ship different words.** By design. Signal: none needed. Response: `meta` -identifies the bytes; the audit note in phase 3 records what this first build contained. - -**The space in the URL.** Signal: `curl` fetching `https://kaikki.org/viwiktionary/Ti%E1%BA%BFng` -and failing. Response: the URL stays percent-encoded in every file, and the pin test checks -that the string contains `Ti%E1%BA%BFng%20Vi%E1%BB%87t`. diff --git a/plans/260908-1653-kaikki-viwiktionary-corpus/phase-03-measure-the-corpus.md b/plans/260908-1653-kaikki-viwiktionary-corpus/phase-03-measure-the-corpus.md deleted file mode 100644 index 89f7812..0000000 --- a/plans/260908-1653-kaikki-viwiktionary-corpus/phase-03-measure-the-corpus.md +++ /dev/null @@ -1,81 +0,0 @@ ---- -phase: 3 -title: "Phase 3: Measure the corpus" -status: completed -priority: P1 -effort: "1h" -dependencies: [1, 2] ---- - -# Phase 3: Measure the corpus - -## Overview - -Record what the first kaikki build contains against today's database: the graph numbers, the -words gained and lost, the POS mix, and bot-versus-bot behaviour. Measurement, not a gate — -no rule removes words any more, so there is nothing to audit by hand. - -## Requirements - -- Functional: a note in this plan directory with the numbers below and the commands that - produced them, plus the `source_sha256` of the file measured so the note is tied to bytes. -- Functional: a diff against the current `data/noitu.db` (copied aside before Phase 2 - overwrites it): shared, gained, lost, with 25 random samples of each. -- Functional: playability on both databases: words, syllables, openers, openers with ≥2 - continuations, dead ends, and the `internal/bot` real-corpus test output. -- Non-functional: every number reproducible from a written command. - -## Architecture - -Pre-measured this session on kaikki's 2026-09-06 file (sha256 `51ddc2fbd73cd7e2…`); the -phase re-measures on whatever the build fetched and explains any delta: - -| | today | kaikki-vi (2026-09-06) | -|---|---|---| -| words | 26,845 | 34,813 | -| syllables | 5,709 | 6,081 | -| openers ≥2 continuations | 2,787 | 3,163 | -| dead-end syllables | 1,551 | 1,594 | -| shared / gained / lost | | 25,392 / 9,421 / 1,453 | - -POS of the 41,507 distinct words: noun 16,219 · verb 9,355 · adj 6,698 · name 5,180 · -unknown 4,652 · adv 1,024 · phrase 392 · proverb 322 · character 198. `name` and `unknown` -are kept; the log prints the tally so a future decision has the numbers. - -## Related Code Files - -- Create: `plans/260908-1653-kaikki-viwiktionary-corpus/measurement.md` -- Read only: the pre-switch `data/noitu.db` copy, the new build, `server/internal/bot` - real-corpus tests - -## Implementation Steps - -1. Before Phase 2 overwrites it, copy today's `data/noitu.db` to a scratch path. -2. After the first `make dict`, record `meta.source_sha256`, `source_rows`, - `source_fetched_at` and the build log's reject and POS lines. -3. Compute the diff and the five graph numbers for both databases (sqlite, one script; keep - the script text in the note). -4. Run `go test ./internal/bot/ -run RealCorpus -v` against both databases (swap the file - under `data/noitu.db`, restore afterwards) and record the game-length and win-rate lines. -5. Look at the 1,453 lost words: they are words in the 2018 scrape that Wiktionary has since - deleted or renamed. Sample 25, note what they look like (expected: misspellings, moved - pages, deleted junk). No action unless the sample is mostly real vocabulary. - -## Success Criteria - -- [x] `measurement.md` exists with the tables, samples, bot lines, commands and the source - SHA-256. -- [x] Words > 30,000; syllables, openers and ≥2-continuation counts all above today's. -- [x] Bot real-corpus tests pass on the new database; easy-vs-easy game length is not - shorter than today's 12.9 moves. -- [x] The lost-word sample is characterised in one paragraph. - -## Risk Assessment - -**The fetched file differs from the one measured today.** Certain over time. Signal: counts -off from the table above. Response: record the new numbers and the SHA-256; the deltas are -the point of the note, not a failure. - -**Dead ends rise slightly (1,551 → 1,594).** More words bring more rare final syllables. -Signal: already known. Response: the ratio of dead ends to syllables falls (27.2% → 26.2%), -so the graph is denser, not sparser; record it and move on. diff --git a/plans/260908-1653-kaikki-viwiktionary-corpus/phase-04-relicense-to-4-0.md b/plans/260908-1653-kaikki-viwiktionary-corpus/phase-04-relicense-to-4-0.md deleted file mode 100644 index 77a7b4c..0000000 --- a/plans/260908-1653-kaikki-viwiktionary-corpus/phase-04-relicense-to-4-0.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -phase: 4 -title: "Phase 4: Relicense to 4.0" -status: completed -priority: P1 -effort: "1h" -dependencies: [3] ---- - -# Phase 4: Relicense to 4.0 - -## Overview - -Current Wiktionary text is CC BY-SA 4.0, so the derived database is too. Restore the 4.0 -legal code, rewrite the attribution chain for a source with no commit to cite, and update -every place that says 3.0 — in the same commit as the first kaikki-built database. - -## Requirements - -- Functional: `data/LICENSE` is the CC BY-SA 4.0 legal code, byte-identical to the file this - repository shipped before commit `39ef457` (`git show e19b083:data/LICENSE`). -- Functional: `NOTICE` section 2 says 4.0 and names Wiktionary tiếng Việt as the original - work and kaikki.org/wiktextract as the extraction; section 1 byte-identical. -- Functional: `data/ATTRIBUTION.md` records the source URL, that the file is refreshed - weekly and unpinned, that the exact bytes of any build are identified by - `meta.source_sha256`, the three attribution links, and the modifications list. -- Functional: the in-game footer links to the 4.0 deed and says 4.0; the e2e assertion - matches. -- Non-functional: no overclaiming about dates — the attribution names the dump date only as - "the Wikimedia dump kaikki extracted at fetch time; see `source_fetched_at`", not a fixed - date that goes stale. - -## Architecture - -Attribution chain, all three named: - -``` -Wiktionary tiếng Việt contributors authors, CC BY-SA 4.0 - → wiktextract / kaikki.org (Tatu Ylonen) extraction; cite LREC 2022 as kaikki requests - → this project filter + index, data/noitu.db, CC BY-SA 4.0 -``` - -Modifications list (renumbered): 1 language selection (`lang_code = vi`), 2 length filter, -3 content filter, 4 normalization, 5 spelling aliases, 6 added columns and tables, -7 deduplication, 8 dropped fields (senses, translations, POS, categories — everything but -the word form). - -## Related Code Files - -- Modify: `data/LICENSE` — restore 4.0 text from git history -- Modify: `NOTICE` — section 2 heading, source lines, share-alike sentence -- Modify: `data/ATTRIBUTION.md` — full rewrite of Source, Modifications 1 and 8, reproduce - block (no checksum step) -- Modify: `README.md` — licence table row, dictionary-source paragraph -- Modify: `Dockerfile`, `docs/deployment.md`, `.github/workflows/ci.yml` — 3.0 → 4.0 in comments -- Modify: `server/cmd/build-dictionary/main.go` doc comment, `server/cmd/noitu-server/main.go` - and `server/internal/dictionary/store.go` comments, `store_test.go` seed strings -- Modify: `web/src/lib/components/AttributionFooter.svelte` (deed URL, comment), - `web/src/lib/i18n/vi.js` (`attributionLicense`), `web/e2e/bot-game.spec.js` (regex) -- Verify unchanged: `.github/workflows/ci.yml` required-files check - -## Implementation Steps - -1. `git show e19b083:data/LICENSE > data/LICENSE`; confirm the first lines read - "Attribution-ShareAlike 4.0 International". -2. Rewrite `NOTICE` section 2 source lines; leave the share-alike paragraph's meaning, change - the version. -3. Rewrite `data/ATTRIBUTION.md` as described. State plainly: "The file is not pinned. Each - build fetches kaikki's current export; the SHA-256 and row count of the file a given - database was built from are recorded in its `meta` table." -4. `grep -rn "3\.0\|by-sa/3" --exclude-dir=plans .` and fix every hit; then the same for - `undertheseanlp`, `DICT_SHA256`, `--merged`, `--sources`. -5. Regenerate `data/noitu.db`; confirm `meta.source_license` says 4.0 and matches the files. -6. Run the full verification set from the previous plan: Go vet/test, web check/test, both - image builds with the CI file checks. - -## Success Criteria - -- [x] `data/LICENSE` is the 4.0 text; `git diff e19b083 -- data/LICENSE` is empty. -- [x] No file outside `plans/` says CC BY-SA 3.0 or names undertheseanlp. -- [x] `NOTICE` section 1 has no diff. -- [x] `ATTRIBUTION.md` names all three links and states the unpinned-fetch policy. -- [x] `meta.source_license` in the built database says 4.0. -- [x] All verification suites green; the commit contains both the licence files and the - builder change so no intermediate tree mixes 3.0 text with 4.0 data. - -## Risk Assessment - -**A stale 3.0 or undertheseanlp claim survives.** Signal: the grep in step 4 finds one after -the phase is called done. Response: grep is a step for that reason; run it last. - -**The footer wording.** `giấy phép CC BY-SA 3.0` → `4.0` is one string, but the e2e regex -and the deed URL must change with it. Signal: e2e footer test fails in CI. Response: all -three edits are listed above; do them together. diff --git a/plans/260908-1653-kaikki-viwiktionary-corpus/plan.md b/plans/260908-1653-kaikki-viwiktionary-corpus/plan.md deleted file mode 100644 index 17f5f89..0000000 --- a/plans/260908-1653-kaikki-viwiktionary-corpus/plan.md +++ /dev/null @@ -1,142 +0,0 @@ ---- -title: "Kaikki viwiktionary corpus" -description: "Replace the 2018 undertheseanlp wiktionary rows with kaikki.org's current extraction of Wiktionary tiếng Việt, fetched unpinned at build time, and move the data licence to CC BY-SA 4.0" -status: completed -priority: P1 -effort: "~1d" -tags: [dictionary, data, licensing, build] -created: 2026-09-08 -blockedBy: [] -blocks: [] ---- - -# Kaikki viwiktionary corpus - -## Overview - -`data/noitu.db` is built from the `wiktionary` rows of `undertheseanlp/dictionary`: a -2018-12-10 scrape of vi.wiktionary.org, 26,845 words after filtering, pinned by commit and -SHA-256 (plan `260908-1525-dictionary-corpus-switch`, completed today). It is the same -website eight years staler than it needs to be. - -This plan switches the source to **kaikki.org's extraction of Wiktionary tiếng Việt** — the -`Tiếng Việt` language file of the `viwiktionary` edition, produced by wiktextract from the -monthly Wikimedia dump (currently 2026-09-01) and refreshed roughly weekly. Same authors, -same website, current text. Measured through the real `build-dictionary` filter this -session: **34,813 words**, a near-superset of today's corpus (25,392 shared, 9,421 gained, -1,453 lost) and 96% of the raw dump (36,200). - -Two owner decisions shape the plan: - -- **viwiktionary only.** kaikki's English-Wiktionary Vietnamese file (21,244 words, marked - deprecated by kaikki) is not used. The game is Vietnamese; one edition, one attribution. -- **No pin — fetch the newest file every build.** kaikki publishes rolling files with no - archived snapshots, so a checksum pin would break weekly. The owner chose freshness over - reproducibility. The build therefore has to *fail loudly* on a bad download (HTTP error, - malformed JSON, too few words) and *record what it actually got* (SHA-256 and row count in - `meta`) so any shipped database can still be traced to exact bytes. - -Consequence: the current text of Wiktionary is CC BY-SA **4.0** (Wikimedia moved from 3.0 -on 2023-06-29), so `data/noitu.db` returns to 4.0. The 3.0 chosen this afternoon only ever -applied to the 2018 snapshot. - -Evidence: `plans/reports/research-260908-1529-viwiktionary-dump-measured.md` (dump and -kaikki-en measurements) plus this session's measurement of the kaikki-vi file, recorded in -[phase 3](./phase-03-measure-the-corpus.md). - -## What changes, in numbers - -| | today | after | -|---|---|---| -| words | 26,845 | **34,813** (30,231 if `name` POS were dropped — it is not) | -| syllables | 5,709 | 6,081 | -| syllables with ≥2 continuations | 2,787 | 3,163 | -| dead-end syllables | 1,551 | 1,594 | -| source | 4.8 MB JSONL, 2018, pinned | **62 MB JSONL, current, unpinned** | -| data licence | CC BY-SA 3.0 | **CC BY-SA 4.0** | -| `--min-words` floor | 20,000 | **30,000** | -| build reproducible | yes | **no** — traceable via `meta.source_sha256` | - -## The source - -``` -URL https://kaikki.org/viwiktionary/Tiếng Việt/kaikki.org-dictionary-TiếngViệt.jsonl - (percent-encoded: /viwiktionary/Ti%E1%BA%BFng%20Vi%E1%BB%87t/kaikki.org-dictionary-Ti%E1%BA%BFngVi%E1%BB%87t.jsonl) -Size 62,288,004 bytes on 2026-09-06 — 44,564 rows, 41,507 distinct words, all lang_code "vi" -Row {"word": "trở thành", "pos": "verb", "lang_code": "vi", "lang": "Tiếng Việt", "senses": [...], ...} -``` - -Only `word` and `lang_code` are read. `pos` is counted for the build log but never filters: -the owner's earlier decision that capitalization does not remove words carries over to the -`name` POS (5,138 name-only words). 5,475 words carry uppercase and 229 lowercase forms -occur twice; `accept()` lowercases and dedupes as it always has. - -## Decisions taken - -- **One corpus input mode.** `--merged`/`--sources` (undertheseanlp) is replaced by - `--kaikki`, not kept beside it. Two readers for one database is dead weight, as the SQLite - path was. The fixture path `--words` stays. -- **The floor moves to 30,000.** 34,813 measured; a truncated or reshaped download must - fail, and 30,000 is the highest round number with a comfortable margin. -- **Provenance is measured, not asserted.** The reader hashes the file as it streams it and - writes `source_sha256`, `source_rows` and `source_fetched_at` into `meta`. There is no - commit to record, so this is what "which bytes" means from now on. -- **Attribution names three links.** Wiktionary tiếng Việt contributors (authors, CC BY-SA - 4.0), wiktextract/kaikki.org by Tatu Ylonen (extraction; kaikki asks for the LREC 2022 - citation), and this project. No GitHub intermediary any more. -- **`verify-dict` goes.** There is nothing to verify against. The - Makefile/Dockerfile agreement test keeps guarding the URL and drops the checksum clauses. - -## Phases - -| # | Phase | Status | -|---|-------|--------| -| 1 | [Read the kaikki file](./phase-01-start.md) | Pending | -| 2 | [Fetch the newest file](./phase-02-fetch-the-newest-file.md) | Pending | -| 3 | [Measure the corpus](./phase-03-measure-the-corpus.md) | Pending | -| 4 | [Relicense to 4.0](./phase-04-relicense-to-4-0.md) | Pending | - -Phase 4 lands in the same commit as the first kaikki-built `noitu.db`: a 4.0-derived -database in a tree whose `NOTICE` says 3.0 is the one ordering mistake with a legal -consequence. Phase 3 is measurement, not a gate — there is no drop rule left to audit. - -## Success criteria - -- [x] `make fetch-dict && make dict` downloads kaikki's current file and builds - `data/noitu.db` with **more than 30,000 words**; a truncated or non-JSON download fails - the build with a message naming the cause. -- [x] `meta` records `source_url`, `source_sha256`, `source_rows`, `source_fetched_at` and - `source_license = CC BY-SA 4.0`; no `source_commit`, `sources_kept` or - `sources_excluded` remain. -- [x] `data/LICENSE` is the CC BY-SA 4.0 legal code; `NOTICE`, `data/ATTRIBUTION.md`, README, - Dockerfile, deployment doc, the builder's licence string and the in-game footer all say - 4.0 and credit Wiktionary tiếng Việt contributors and wiktextract/kaikki.org. -- [x] Playability measured and recorded against today's database, including bot game - lengths; nothing regresses below today's numbers. -- [x] `go vet ./... && go test ./...`, `npm run check && npm test`, the Docker image with the - CI file checks and `FIXTURE_DICT=1` are green. e2e: not runnable on this machine (Playwright Chromium download fails); the one changed assertion was updated by hand, CI verifies. -- [x] No file outside `plans/` names undertheseanlp, `--merged`, `--sources` or `DICT_SHA256`. - -## Review - -Code review (2026-09-08) returned DONE_WITH_CONCERNS; every finding was applied: `fetch-dict` -downloads to a `.part` name and renames on success so an interrupted fetch never feeds the -next `make dict`; a failed `Stat` is an error rather than a zero `source_fetched_at`; -`builder_version` bumped to 3 for the changed meta contract; `.gitignore` covers -`data/*.jsonl`; `ATTRIBUTION.md` states kaikki's redistribution terms; bare JSON literals -are malformed rather than "foreign-language"; the non-EOF read-error line number is exact; -the CI leak guard matches any `.jsonl`; byte-exact reader tests cover a missing final -newline, an HTML error page, a mid-line cut and a bare literal. The reviewer's residual -note stands: with no pin, the build trusts kaikki.org over TLS, and the recorded SHA-256 is -the compensating control; blast radius is game content only. - -## Open questions - -- kaikki's weekly refresh means two builds a week apart can differ. Accepted by the owner; - `source_sha256` in `meta` is the answer to "which words did this image ship". Should a - release ever need to be rebuilt bit-for-bit, the fallback is to keep the fetched file as a - release asset — a Makefile variable change, not a redesign. -- The `unknown` POS bucket (4,616 words such as `tình báo`, `trọc phú`, `hội quán`) is real - vocabulary the vi extractor could not label. Kept, like everything else. - - diff --git a/plans/260908-2056-wiktionary-dump-corpus-and-meanings/measurement.md b/plans/260908-2056-wiktionary-dump-corpus-and-meanings/measurement.md deleted file mode 100644 index ab50835..0000000 --- a/plans/260908-2056-wiktionary-dump-corpus-and-meanings/measurement.md +++ /dev/null @@ -1,251 +0,0 @@ ---- -title: "Measurement: first dump-built database against the kaikki one" -date: 2026-09-08 -plan: 260908-2056-wiktionary-dump-corpus-and-meanings -status: complete ---- - -# Measurement: the dump-built database - -Both databases were built on 2026-09-08 on the same machine with the same `accept()` filter. -The kaikki one was built with the last kaikki-era builder (`builder_version` 4) from the -export kaikki served that afternoon; the dump one with this plan's builder (`builder_version` -5) from the `latest/` dump, which was the 2026-09-01 run. - -## The file measured - -| | | -|---|---| -| URL | `https://dumps.wikimedia.org/viwiktionary/latest/viwiktionary-latest-pages-articles.xml.bz2` | -| Resolved to | the 2026-09-01 run (`source_fetched_at` 2026-09-01T11:52:49Z, the file's own mtime kept by `curl -R`) | -| Size | 63,513,513 bytes | -| SHA-256 | `ed66c932f535b0b362d1141c02273b02ed01c4101e856378bd15e88d277bb8d9` (matches the research report's hash of the dated file) | -| `source_pages` | 43,011 | -| Decompressed | 505,982,744 bytes | - -## Build log - -``` -pages 391543, in the main namespace 349461, redirects skipped 3237, without a Vietnamese section 303213 -Vietnamese sections: legacy {{-vie-}} 35884, new == {{langname|vi}} == 7127, pages with both 0, titles merged 131 -parts of speech: noun 15890, verb 8983, adj 6285, place 3258, pr-noun 1666, n 987, adv 689, phrase 589, idiom 541, v 503, adjc 499, proverb 333, proper noun 159, interj 131, … -headings without a label: cụm động từ 2, giải thích 1, định nghĩa 1 -codes that ended a legacy section, commonest: tyz 256, eng 152, mtq 147, nut 49, vi-m 47, nuo 39, tyj 29, kpm 16, mlc 14, tou 14, aav-qal 13, vie-m 12, jra 10, sqi 10, bdq 9 -definitions kept 40937 (cut at 200 characters: 515), dropped as empty after stripping 271 -templates dropped whole, commonest: alternative spelling of 77, vi-alternative spelling of 40, senseid 37, vie alternative spelling of 36, misspelling of 34, syn of 32, mention 18, en 13, zh 13, synonym of 11 - rejected 4: contains a digit - rejected 155: contains punctuation - rejected 6359: fewer than 2 syllables - rejected 162: no Vietnamese letters - rejected 303213: not a Vietnamese-language entry -accepted 36200 distinct words, 35062 with a meaning, (43011 pages, sha256 ed66c932f535b0b362d1141c02273b02ed01c4101e856378bd15e88d277bb8d9) in 32s -generated 1736 spelling aliases (558 skipped as ambiguous or already real words) -wrote data/noitu.db -``` - -**Wall time: 21–32 s** for the whole build on a laptop, Go's pure-Go bzip2 included. The -two-minute risk did not materialise; the `bzip2 -dc` fallback was not needed. - -The counters reconcile: 349,461 main-namespace pages − 3,237 redirects − 303,213 without a -Vietnamese section = 43,011 = `source_pages`. The definition counters are taken after -`accept()`, so they describe words that land: 40,937 definitions kept against 40,842 rows, -the difference being senses past the cap on the 131 merged title pairs. The "codes that -ended a legacy section" line lists only language codes (tyz, eng, mtq, nut, …), which is the -check that no heading code is missing from the maps and cutting sections short. - -The legacy count (35,884) is one page short of the research report's 35,885 because one page -opens its Vietnamese section after a level-2 heading of another dialect and is read as that -one; the new-dialect count differs by one for the same reason (7,127 vs 7,128). "Pages with -both" is zero, so the merge path for mixed pages stays untested against real data. - -## Graph metrics, old vs new - -| metric | kaikki (v4) | dump (v5) | change | -|---|---|---|---| -| words | 34,813 | **36,200** | +1,387 | -| syllables | 6,081 | 6,172 | +91 | -| syllables that open a word | 4,487 | 4,600 | +113 | -| …with ≥ 2 continuations | 3,163 | 3,246 | +83 | -| dead-end syllables | 1,594 | 1,572 | −22 | -| aliases | 1,664 | 1,736 | +72 | -| database size on disk | 2.2 MB | 6.9 MB | meanings text | - -No metric regresses. - -## Corpus diff - -| | | -|---|---| -| shared | **34,813** — every kaikki word is in the dump build | -| gained | 1,387 | -| lost | **0** | - -The first build lost 15 words (`tây tạng`, `thượng hải`, `nam kinh`, …). All were pages that -run `{{-vie-}}{{-pron-}}{{vie-pron|…}}{{-place-}}` together on one line, which the scanner -read as no section at all. Fixed in phase 1 (headings are read from a line that starts with -`{{-`, several per line); the second build lost none. - -Gained, 25 at random: nghiến ngấu, hồn ai nấy giữ, pháo đùng, trèo cao té đau, ngồi rồi, tối -như hũ nút, tiêu tức, kinh kỳ, rón rón, tự phê bình, vàng hương, trơ mắt ếch, thìa khóa, qua -cầu rút ván, tinh khí, rộng chân rộng cẳng, say lử cò bợ, miệng ăn, sảo thai, phỉ dạ, vắng -như chùa bà banh, cọc tìm trâu, óng a óng ánh, lồm lộp, phơi phóng. Reduplicatives and -idioms, as the research predicted. - -## Bot vs bot, 60 games each - -Run with `go test ./internal/bot/ -run RealCorpus -v -count=1` against each database copied -to `data/noitu.db`. (The kaikki database needed an empty `meanings` table and a -`meaning_count` row added to a scratch copy: the v5 store refuses a v4 file, by design.) - -| | kaikki | dump | -|---|---|---| -| hard beats easy | 100% (3.4 moves) | 95% (3.5 moves) | -| medium beats easy | 88% | 82% | -| hard beats medium | 60% (2.9 moves) | 72% (4.1 moves) | -| easy vs easy, game length | 16.8 moves | 13.7 moves | -| hard decision latency | n=74 p95 2.6 ms max 3.4 ms | n=68 p95 2.5 ms max 4.0 ms | - -Both pass the ladder test's thresholds. The differences are within what a 60-game sample -moves between runs; the corpus is 4% larger and the graph slightly better connected. - -## Meanings - -| | | -|---|---| -| `meaning_count` (senses) | 40,842 | -| `words_with_meaning` | 35,062 of 36,200 = **96.9%** (floor in `verify`: 60%) | -| senses per word | 1: 30,454 · 2: 3,773 · 3: 583 · 4: 167 · 5 (the cap): 85 | -| definitions cut at 200 characters | 587 | -| definitions dropped as empty after stripping | 942 | -| senses with an empty part-of-speech label | 4,849 = **11.9%** | - -Labels, by sense: danh từ 14,471 · động từ 7,804 · tính từ 5,839 · (empty) 4,849 · địa danh -3,702 · danh từ riêng 1,859 · phó từ 758 · cụm từ 544 · thành ngữ 446 · tục ngữ 329 · thán từ -112 · đại từ 47 · liên từ 44 · số từ 17 · giới từ 11 · tiền tố 7 · trợ từ 3. - -The empty share is above the plan's 10% target, but the unmapped-heading list has been worked -through: every code seen more than once is mapped (see below). What remains empty is the -research report's "no-pos" class — pages whose Vietnamese section has definitions under no -part-of-speech heading at all (`sun phát`, `kiến vàng`, `long đong` in the sample). Nothing -in the wikitext gives those a label. - -**Headings without a label, by frequency:** cụm động từ 2, classifier 1, det 1, giải thích 1, -prop 1, reading 1, tiếng anh 1, tyj 1, định nghĩa 1, đồng nghĩa khác âm 1. Nothing worth a -map entry. - -**Templates dropped whole, ten commonest:** `vi-han form of` 222, `vi-nom form of` 163, -`alternative spelling of` 96, `senseid` 79, `rfdef` 63, `abbreviation of` 48, -`vi-alternative spelling of` 45, `vie alternative spelling of` 44, `vie-han form of` 41, -`misspelling of` 36. The "form of" and "spelling of" family is the one worth teaching next: -a definition that is only `{{vi-alt sp|hoá thạch}}` strips to nothing today, and `hóa thạch` -is one of the 1,138 words without a meaning for that reason. `rfdef` is a request for a -definition and correctly yields none. - -### Deviations from the plan's label map, decided in phase 1 from the frequency tables - -- `pron` on this wiki is **pronunciation** (36,533 legacy headings, 5,501 new), not pronoun. - The pronoun code is `pronoun` (158 + 141), also `per-pronoun`. The plan's `pron → đại từ` - would have labelled 36,000 pronunciation sections as pronouns. -- The new dialect uses `{{section|code}}` (3,506 headings) as well as `{{ĐM|code}}`, and the - shorthand codes `n` (647 + 9,488 wiki-wide) and `v` (353 + 3,774) beside `noun`/`verb`. -- `{{-dfn-}}` (4,615) is a "definitions" heading placed *under* a part-of-speech heading, so - it is transparent to the label rather than a heading of its own. -- Bare headword lines `{{vi-noun}}`, `{{vie-noun}}`, `{{vi-pr-noun}}` set the label when - their code is a part of speech; `{{vi-pron}}`, `{{vi-etym-sino}}` and kin do not. -- Two definition-shaped templates beyond the plan's list were taught because they top the - definition-line template counts: `{{nhãn|…}}`/`{{context|…}}`/`{{term|…}}` as label - spellings, `{{n-g|…}}` as a non-gloss definition, and `{{see-entry|x}}`/`{{like-entry|x}}` - (13,122 + 1,038 definition lines wiki-wide) rendered as `Xem x`. -- Definition lines without a space after `#` (`#{{term|cái cân}} Đồ dùng…`) are accepted; - only `##`, `#:`, `#*`, `#;` are sub-items. - -## The 50-sense sample - -Drawn with `SELECT word, pos, gloss FROM meanings ORDER BY RANDOM() LIMIT 50` from the -first build and read by hand. Verdicts: **fine 43, terse 5, wrong 2.** - -| # | word | pos | gloss | verdict | -|---|---|---|---|---| -| 1 | quý tộc | danh từ | Họ dòng sang. | terse | -| 2 | khuyến mại | động từ | Xúc tiến việc mua bán hàng hóa, cung ứng dịch vụ bằng cách dành cho khách hàng những lợi ích nhất định. | fine | -| 3 | bất đắc dĩ | tính từ | Không có sự lựa chọn khác. | fine | -| 4 | hắc điếm | danh từ | (từ cổ) Quán trọ, khách sạn, nơi tạm trú (có thể do kẻ xấu lập ra nhằm cướp của, giết người khi có dịp). | fine | -| 5 | đèn vách | danh từ | Đèn dầu hoả treo trên vách nhà. | fine | -| 6 | chuẩn cơm mẹ nấu | cụm từ | Món hợp khẩu vị, ăn mãi không ngán. | fine | -| 7 | tác ác | động từ | Làm việc ác. | fine | -| 8 | ỉ ê | tính từ | Từ gợi tả tiếng khóc nhỏ, dai dẳng và ỉ eo một cách khó chịu (thường nói về trẻ con) | fine | -| 9 | dơ bẩn | tính từ | (Phương ngữ) Xem nhơ bẩn | fine (cross-reference) | -| 10 | tốt bộ | | Chỉ đẹp có bề ngoài. | fine, no label | -| 11 | hãm hại | động từ | Làm hại, giết chết bằng thủ đoạn ám muội. | fine | -| 12 | thuần hậu | tính từ | Chất phác hiền hậu. | fine | -| 13 | phúc bạc | | Phúc mỏng, ít phúc (không phải bạc là trắng, dù tác giả có ý đối với chữ má đào). | fine, no label | -| 14 | chặt chẽ | tính từ | Có quan hệ khăng khít, gắn kết với nhau. | fine | -| 15 | long đong | | Vất vả, nay đây mai đó, hay gặp nhiều rủi ro. | fine, no label | -| 16 | hồng bì | danh từ | Thứ cây cùng họ với cam, quít, quả nhỏ, da vàng, có lông nhung, vị chua ngọt. | fine | -| 17 | thế chiến i | danh từ riêng | Xem Chiến tranh thế giới thứ nhất. | fine (cross-reference) | -| 18 | khuyên nhủ | động từ | Khuyên bảo ân cần. | fine | -| 19 | thường thới hậu a | địa danh | Một xã thuộc huyện Hồng Ngự, tỉnh Đồng Tháp, Việt Nam. | fine | -| 20 | gạo sen | danh từ | Hạt trắng hình hạt gạo, ở đầu nhị đực của hoa sen, dùng để ướp chè. | fine | -| 21 | kèn trống | danh từ | Kèn và trống thường sử dụng trong đám ma. | fine | -| 22 | lộng quyền | động từ | Làm việc vượt quá quyền hạn của mình, lấn cả quyền hạn của người cấp trên. | fine | -| 23 | quay đĩa | danh từ | (Khẩu ngữ) máy quay đĩa (nói tắt) | fine | -| 24 | tam giáo | danh từ | (Id.). Ba thứ đạo ở Trung Quốc thời trước. | terse (a source abbreviation survived) | -| 25 | điều dưỡng viên | danh từ | Người phụ trách công tác điều dưỡng, … cho đến phục hồi, trị… | fine (cut at 200) | -| 26 | bán tống bán tháo | cụm từ | (Khẩu ngữ) Xem bán đổ bán tháo | fine (cross-reference) | -| 27 | kiểu cách | danh từ | Kiểu mẫu và cách thức. | fine | -| 28 | hiểm ác | tính từ | Ác một cách ngấm ngầm. | fine | -| 29 | ca nô | danh từ | Thuyền máy cỡ nhỏ, mạn cao, có buồng máy, buồng lái, dùng chạy trên quãng đường ngắn. | fine | -| 30 | bút pháp | danh từ | (cũ) Phong cách viết chữ Hán. | fine | -| 31 | thuần việt | tính từ | Có nguồn gốc từ người Việt. | fine | -| 32 | sun phát | | Muối của a-xít sun-phu-rích. | fine, no label | -| 33 | kiến vàng | | Loài kiến nhỏ, màu vàng, đốt đau. | fine, no label | -| 34 | trinh phú | địa danh | Một xã thuộc huyện Kế Sách, tỉnh Sóc Trăng, Việt Nam. | fine | -| 35 | lạnh gáy | tính từ | Xem lạnh người. | fine (cross-reference) | -| 36 | giải phóng | động từ | Làm cho được tự do, cho thoát khỏi địa vị nô lệ hoặc tình trạng bị áp bức, kiềm chế, ràng buộc. | fine | -| 37 | đẹp duyên | động từ | (kiểu cách) Xem kết duyên | fine (cross-reference) | -| 38 | thanh đạm | tính từ | (Ẩm thực) Giản dị, không có những món cầu kì hoặc đắt tiền. | fine | -| 39 | suy tàn | động từ | Ở trạng thái suy yếu và tàn lụi, không còn sức sống. | fine | -| 40 | tràng thạch | danh từ | (Địa lý học). | **wrong** — the definition after the label was a template the stripper dropped | -| 41 | toàn bộ | danh từ | Tất cả các phần, các bộ phận của một chỉnh thể. | fine | -| 42 | bát tiên | danh từ riêng | Tám vị tiên là Hán Chung Ly, … hay được vẽ trên màn trướng. | fine | -| 43 | ba mũi giáp công | danh từ | Tiến công bằng ba hình thức kết hợp: quân sự, chính trị và binh vận. | fine | -| 44 | sở ước | | Điều mình mong được. | fine, no label | -| 45 | giao diện lập trình ứng dụng | danh từ | (programming) Đặc tả quy định cách … | terse (English label from a pasted template) | -| 46 | ngộ gió | | Bị cảm vì gặp gió. | fine, no label | -| 47 | ảo tung chảo | tính từ | Cảm giác khó tin. | terse | -| 48 | nhóm bếp | | Đốt lửa cho củi bắt đầu cháy trong bếp. | fine, no label | -| 49 | trần hy tăng | danh từ riêng | Xem Trần Bích San | fine (cross-reference) | -| 50 | cung ứng | động từ | Cung cấp đáp ứng nhu cầu, thường là của sản xuất, hoặc của hành khách. | fine | - -Two wrong out of fifty, for different reasons; well under the one-in-five threshold that -would have sent the stripper back to phase 1. The one systematic gap is a label followed by -a template-only definition (#40), which leaves a bare `(label).` — a follow-up for the -stripper, template by template. - -## Words without a meaning - -1,138 words (3.1%). Three read at random: - -- `hóa thạch` — one definition, `# {{vi-alt sp|hoá thạch}}`, an alternative-spelling - template the stripper does not know. -- `lóng lánh`, `khoái trá` — the page has only pronunciation, paronyms and a `{{-see-}}` - section with `{{like-entry|…}}` as a bullet, not a `#` line. No definition to carry. - -## Verification run alongside this measurement - -- `go vet ./... && go test ./... -race`: green. -- `npm run check && npm test`: green (183 tests). -- `npm run test:e2e` and the two Docker image variants: see the plan's session log. - -## After code review - -The review (`plans/reports/code-reviewer-260908-2210-wiktionary-dump-corpus-and-meanings.md`) -found no critical issue; its findings were fixed in the same working tree: the bot-game e2e -assertion now reads the drawn opening's sense from the fixture list instead of assuming -`danh từ`; the chain-row styling is scoped to the outer list so the sense list renders as a -numbered list; the stripper drops format characters (bidi overrides, zero-width spaces) as -well as controls; a pre-v5 database is refused with a message that says to rebuild; a lone -short template parameter is no longer mistaken for a language code; the definition counters -are taken after `accept()`; section-ending codes are tallied; the meanings lookup happens once -per move; `aria-controls` is set only while the panel exists. The numbers above are from the -build after those fixes. diff --git a/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-01-start.md b/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-01-start.md deleted file mode 100644 index e9010cd..0000000 --- a/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-01-start.md +++ /dev/null @@ -1,192 +0,0 @@ ---- -phase: 1 -title: "Phase 1: Read the dump" -status: completed -priority: P1 -effort: "6h" -dependencies: [] ---- - -# Phase 1: Read the dump - -## Overview - -Replace the kaikki JSONL reader in `build-dictionary` with a streaming reader for the -Wikimedia `pages-articles.xml.bz2` dump that yields, per Vietnamese-section page, the title -for `accept()` and the stripped definition lines for the meanings table. - -## Requirements - -- Functional: `--dump ` reads a bzip2-compressed MediaWiki XML export. Pages with - `ns != 0` or a `` element are skipped and counted. Every other page's `` - and latest `<revision><text>` are handed to the section scanner. -- Functional: the scanner finds the Vietnamese section in either dialect — `{{-vie-}}` or a - level-2 heading whose text is `{{langname|vi}}` — and stops at the section's end: for - legacy pages, the next `{{-xxx-}}` template whose code is not in the section-code set; for - new pages, the next level-2 heading. Pages with no Vietnamese section are counted, not - read. Pages with a section in *both* dialects are read once, first dialect wins, counted. -- Functional: the title goes to `accept()` unchanged from today. Part of speech labels seen in - the section (`{{-noun-}}`, `{{ĐM|noun}}`, `{{vi-noun}}` and kin, `pr-noun`/`place` - included) are tallied for the log and never filter. -- Functional: within the section, every line starting with `# ` (exactly one `#`, then a - space — not `#:`, `#*`, `##`) is a definition. It is stripped (see Architecture), capped at - 200 runes with an ellipsis, and appended to the page's senses up to 5. Empty results are - dropped. Senses are kept in page order. -- Functional: each sense carries the part of speech of the heading it sits under, as a - Vietnamese label from a fixed map keyed by the heading code — the same codes in both - dialects (`{{-noun-}}`, `{{ĐM|noun}}`, `{{vi-noun}}` → `danh từ`). Starting map: `noun` danh - từ, `verb` động từ, `adj` tính từ, `adv` phó từ, `pr-noun` danh từ riêng, `place` địa danh, - `pron` đại từ, `num` số từ, `conj` liên từ, `prep` giới từ, `intj` thán từ, `part` trợ từ, - `phrase` cụm từ, `idiom` thành ngữ, `prov` tục ngữ, `abbr` viết tắt, `prefix` tiền tố, - `suffix` hậu tố, `char` chữ. A definition under a heading outside the map, or before any - heading, has an empty label and is kept; unmapped codes are counted for the log. Step 1's - frequency list decides what else the map needs. (Validation session 1.) -- Functional: two pages whose titles normalize to the same word (`Việt Nam` and `việt nam` - both being real pages) merge: the word is one entry, senses concatenate in page order and - the cap applies to the union. -- Functional: provenance is measured during the one streaming pass: SHA-256 of the compressed - bytes as read, `source_pages` (pages with a Vietnamese section, redirects excluded), - `source_fetched_at` from the file's modification time. `source_rows` is gone. -- Functional: the build log reports pages seen, ns0 pages, redirects skipped, pages with a - Vietnamese section per dialect, POS tally, definitions kept, definitions dropped as empty - after stripping, and the ten commonest template names dropped whole. -- Functional: `--kaikki` and `kaikki_list.go` and its tests are removed. Exactly one of - `--dump` / `--words` must be given, same rule as today. -- Non-functional: a stream that is not bzip2, XML that ends mid-page, or a `<text>` element - the decoder cannot read is an error naming the page title or byte offset. A count of - Vietnamese-section pages under 20,000 is an error naming the count, distinct from the word - floor, so a parser that silently misses a dialect is caught before `--min-words` is. -- Non-functional: memory is one page at a time. The XML decoder is used token by token; only - `title`, `ns`, `redirect` and `text` of the current page are held. -- Non-functional: wall time under two minutes on the real dump on a laptop. Measure and - record; see Risk. - -## Architecture - -``` ---dump <file.xml.bz2> - │ os.Open → io.TeeReader(f, sha256) → bufio.Reader → compress/bzip2 → xml.NewDecoder - ▼ -for each <page>: - ns==0, no <redirect> ─no─► count, skip - section := vietnameseSection(text) // dialect-aware slice of the wikitext - section == "" ─yes─► count "no Vietnamese section", skip - word, syllables, reason, ok := accept(title) - senses := definitions(section) // []sense{pos, gloss}: stripped, ≤5, ≤200 runes each - pos tally from section headings - if ok: words[word] = entry{…}; meanings[word] = merge(meanings[word], senses) -``` - -`type sense struct { pos, gloss string }` lives in `wikitext.go`; `pos` is the mapped -Vietnamese label or empty. - -Files: - -``` -server/cmd/build-dictionary/ - dump.go readDump: bz2+xml streaming, page loop, provenance, dumpSourceURL const - wikitext.go vietnameseSection, definitions, stripWikitext, posLabels, posLabelMap - dump_test.go byte-exact fixtures: one legacy page, one new-dialect page, a redirect, - a foreign-only page, a truncated stream, a non-bzip2 file - wikitext_test.go table tests for the stripper and the section boundaries -``` - -The section-code set for legacy pages is the list of `{{-xxx-}}` codes that are *headings* -rather than language switches: `etym, pron, noun, verb, adj, adv, pr-noun, place, phrase, -idiom, prov, syn, synonym, ant, trans, ref, reference, see, der, info, num, pron, conj, -prep, intj, part, abbr, char, hanzi, nom, …`. Build it from the real dump in this phase: -extract every `{{-xxx-}}` code, list them by frequency, and classify by hand; the language -codes are 2–3 letters and the section codes are English abbreviations, so the split is -readable. Record the set in `wikitext.go` with the frequency it was seen at. - -The stripper, in order: - -1. Drop `<!-- … -->`, `<ref …>…</ref>`, `<ref … />`, any other tag pair or lone tag. -2. Templates by a depth counter over `{{`/`}}`. For each outermost template, split on `|` - outside nested braces and brackets, take the name: - - `label`, `lb`, `gloss`, `qualifier`, `q` → `(` join of positional params after a leading - `vi` `)`; - - `l`, `vi-l`, `w`, `m`, `link` → the last positional param (display text if any); - - `place` → positional params after `vi`, each with a leading `[a-z]/` removed, joined by - `, `; - - anything else → dropped, name counted. -3. Links: `[[Thể loại:…]]`/`[[Category:…]]` dropped; `[[a|b]]` → `b`; `[[a]]` → `a`; - external `[http… label]` → `label`. -4. `'''`, `''` removed. ` ` and the common entities decoded. -5. Control characters (`unicode.IsControl`) removed; whitespace collapsed; trimmed. A result - that is only punctuation is empty. -6. Longer than 200 runes: cut at the last space before 200 and append `…`. - -## Related Code Files - -- Create: `server/cmd/build-dictionary/dump.go`, `wikitext.go`, `dump_test.go`, - `wikitext_test.go` -- Delete: `server/cmd/build-dictionary/kaikki_list.go`, `kaikki_list_test.go` -- Modify: `server/cmd/build-dictionary/main.go` — package doc, `config.kaikki` → `dump`, - flag text, `run()` switch, `runFromKaikkiList` → `runFromDump`, `entry` gains nothing (the - meanings map travels beside `words` into `finish()` — phase 3 persists it), - `builderVer = "5"` -- Modify: `server/cmd/build-dictionary/main_test.go` — `fixtureSource` writes a small bzip2 - XML dump instead of JSONL (Go has no bzip2 *writer*: commit a tiny `testdata/mini-dump.xml.bz2` - built once with `bzip2`, plus its uncompressed source beside it for review) -- Modify: `server/cmd/build-dictionary/filter.go` — `rejectNotVietnamese` comment now refers - to a page with no Vietnamese section; the reason string can stay - -## Implementation Steps - -1. Download the current dump once locally (phase 2's URL, `curl -fLR`). Write a throwaway - `go run` that streams it and prints every `{{-xxx-}}` code with counts and every level-2 - heading text with counts. Build the section-code set and the dialect markers from that - output; keep the numbers in comments. -2. Write `wikitext.go`: `vietnameseSection`, `posLabels`, `definitions`, `stripWikitext`. - Table-test the stripper on the three samples in `plan.md` plus: nested templates, a - `<ref>` mid-sentence, a link with a category, a definition that is only `{{rfdef|vi}}` - (must be empty), a 300-rune definition (must end in `…` at a word boundary), a legacy - page whose senses sit under `{{-noun-}}` then `{{-verb-}}` (labels `danh từ`, `động từ` in - order), a new-dialect page with `=== {{ĐM|pr-noun}} ===` (label `danh từ riêng`), a - definition under an unmapped heading (empty label, sense kept). -3. Write `dump.go`: the streaming loop, provenance, counters, the error shapes listed under - Non-functional. Test against `testdata/mini-dump.xml.bz2` (six pages, both dialects, a - redirect, an English-only page) and byte-truncated copies of it. -4. Rewire `main.go`; delete the kaikki files; make `go vet ./... && go test ./cmd/...` green. -5. Run against the real dump with `--out /tmp/dump.db`. Read the log: pages per dialect should - be within a few percent of 35,885 / 7,128 for the 2026-09-01 dump; accepted words near - 36,200. Record wall time. Keep this database for phase 6. -6. Sample 50 senses at random from the log (or from the phase-3 table) and read them. Fix any - stripper rule that is clearly wrong across many entries; note the rest for phase 6. - -## Success Criteria - -- [x] `go run ./cmd/build-dictionary --dump ../data/viwiktionary-latest-pages-articles.xml.bz2 --out /tmp/dump.db` - completes; the log shows ~43,000 Vietnamese-section pages, both dialects non-zero, - ~36,200 accepted words, and a definitions-kept count above 30,000. -- [x] The stripper table tests pass on all listed shapes; `# {{rfdef|vi}}` yields no sense; - labels come out in Vietnamese for both dialects and empty for an unmapped heading. -- [x] On the real dump, senses with an empty label are under 10% of all senses, or the - unmapped-code list has been worked through. -- [x] A truncated dump, a non-bzip2 file and a mid-page cut each fail with a message naming - the cause; a fixture with 3 Vietnamese pages fails the 20,000-page check by name. -- [x] `grep -rn kaikki server/` finds nothing. -- [x] Wall time on the real dump recorded in the phase-6 measurement file. - -## Risk Assessment - -**Go's bzip2 is slow.** Signal: step 5 takes longer than two minutes. Response: keep the -reader on an `io.Reader`, add `--dump` acceptance of a plain `.xml` (sniff the two magic -bytes `BZ`), and have the Makefile pipe `bzip2 -dc` in; the Docker `dict` stage on alpine has -`bzip2`. Decide in this phase, not later. - -**The section-code set is incomplete.** A legacy heading code missing from the set is read as -a language switch and truncates the Vietnamese section early: definitions after it are lost, -the word is not. Signal: the definitions-kept count is well below the page count, or the -sample in step 6 shows senses cut off. Response: the step-1 frequency list is the source of -truth; every code seen more than ~20 times must be classified. - -**Definitions in a template the stripper does not know.** The whole sense is dropped. Signal: -"definitions dropped as empty" is large, or a common template name tops the dropped list. -Response: teach the stripper that template if it is a definition-shaped one (`place`, -`label` are the known cases); otherwise accept the loss and record it in phase 6. - -**Both dialects on one page.** Rare; first dialect wins and the page is counted. Signal: the -counter is not small. Response: read both sections and merge senses — a small change to -`vietnameseSection` returning a slice. diff --git a/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-02-fetch-the-dump.md b/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-02-fetch-the-dump.md deleted file mode 100644 index 5642c58..0000000 --- a/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-02-fetch-the-dump.md +++ /dev/null @@ -1,104 +0,0 @@ ---- -phase: 2 -title: "Phase 2: Fetch the dump" -status: completed -priority: P1 -effort: "2h" -dependencies: [1] ---- - -# Phase 2: Fetch the dump - -## Overview - -Point the Makefile, the Docker `dict` stage, the pin test, the CI leak guard and the README's -raw commands at Wikimedia's rolling `latest` dump instead of kaikki's JSONL, keeping the -three URL copies in agreement and a failed download unable to pass as a success. - -## Requirements - -- Functional: `DICT_URL` is - `https://dumps.wikimedia.org/viwiktionary/latest/viwiktionary-latest-pages-articles.xml.bz2`; - `DICT_SRC` is `data/viwiktionary-latest-pages-articles.xml.bz2`. `fetch-dict` uses - `curl -fLR` — `-R` keeps the server's modification time, which is the dump's generation - time and becomes `source_fetched_at` — to a `.part` name renamed on success, as today. -- Functional: `dict` runs `build-dictionary --dump ../$(DICT_SRC)`. -- Functional: the Dockerfile `dict` stage mirrors the URL and the flag; `FIXTURE_DICT=1` is - untouched. `curl` gains `-R` there too. -- Functional: `web/tests/dictionary-source.test.js` asserts the Makefile and Dockerfile URLs - agree, that the URL matches - `^https://dumps\.wikimedia\.org/viwiktionary/latest/viwiktionary-latest-pages-articles\.xml\.bz2$`, - that the builder's `dumpSourceURL` constant in `dump.go` is the same string, and that README - and `data/ATTRIBUTION.md` quote it. -- Functional: the CI leak guard rejects any `\.bz2$` or `\.xml$` inside the image; the - comment says the dump, not the export. `.gitignore` covers `data/*.bz2`, `data/*.bz2.part`, - `data/*.xml` (for the phase-1 fallback) and drops the `.jsonl` lines. -- Functional: README's "Without make" block, Make targets table and Setup section describe - a ~61 MB monthly dump; the `fetch-dict` help line says the same. -- Non-functional: still no resume flag. `latest` can be repointed between two attempts. - -## Architecture - -```make -DICT_URL := https://dumps.wikimedia.org/viwiktionary/latest/viwiktionary-latest-pages-articles.xml.bz2 -DICT_SRC := data/viwiktionary-latest-pages-articles.xml.bz2 - -fetch-dict: - @mkdir -p data - curl -fLR -o $(DICT_SRC).part $(DICT_URL) && mv $(DICT_SRC).part $(DICT_SRC) - -dict: $(DICT_SRC) - cd server && go run ./cmd/build-dictionary --dump ../$(DICT_SRC) --out ../$(DICT_OUT) -``` - -Correctness rests on: `curl -f` plus the `.part` rename (HTTP errors, interrupted -downloads); the bzip2 decoder (a truncated stream errors at EOF); the XML decoder (a cut -mid-page); the 20,000-page check and the 30,000-word floor (content). - -## Related Code Files - -- Modify: `Makefile` — header comment, `DICT_URL`, `DICT_SRC`, `help`, `fetch-dict`, `dict` -- Modify: `Dockerfile` — `dict` stage comment, `ARG DICT_URL`, `curl -fsSLR`, file name, - `--dump` -- Modify: `web/tests/dictionary-source.test.js` — URL pattern, builder file and constant name -- Modify: `.github/workflows/ci.yml` — leak guard pattern and comment; the `image` job's - comment about "the real upstream release" still holds -- Modify: `.gitignore` -- Modify: `README.md` — Setup, Make targets, Without make, Architecture table's dictionary - row wording if it names the export - -## Implementation Steps - -1. Update the Makefile; run `make fetch-dict` on a clean `data/` and check the file's mtime is - the dump's, not now. -2. Mirror in the Dockerfile. -3. Rewrite the pin test; run `npx vitest run tests/dictionary-source.test.js`. Edit one URL - copy alone and confirm it fails. -4. Update the CI leak guard, `.gitignore`, README. -5. `make dict`; then `docker build .` and `docker build --build-arg FIXTURE_DICT=1 .`; run - the CI file checks against the real image and confirm no `.bz2`/`.xml` inside. -6. Point `DICT_URL` at a 404 once and confirm `fetch-dict` fails; feed `dict` a copy of the - dump cut at 40 MB and confirm the build fails naming the cause. - -## Success Criteria - -- [x] `make fetch-dict` downloads ~61 MB with the server's mtime; `make dict` builds >30,000 - words with meanings. -- [x] `web/tests/dictionary-source.test.js` passes and fails when any one copy is edited. -- [x] Both image variants build; the real one passes the CI file checks; nothing matching - `\.bz2$|\.xml$` inside. -- [x] `grep -rniI kaikki --exclude-dir=plans .` finds nothing outside `plans/`. -- [x] A 404 URL and a truncated file each fail the build readably. - -## Risk Assessment - -**`latest/` mid-repoint.** Wikimedia updates the `latest` symlinks file by file as a run -completes. Fetching during the run can return the previous month's file, which is correct, -or in principle a 404 for a moment. Signal: `curl: (22)`. Response: retry; nothing to fix. - -**Dump size grows.** Irrelevant to correctness; the README's "~61 MB" goes stale slowly. -Signal: none. Response: update the number when it is off by a quarter. - -**The Docker build now decompresses bzip2 in Go inside alpine.** Same code, same speed as -locally. If phase 1 chose the `bzip2 -dc` fallback, the stage needs `apk add bzip2` and the -pipe; the leak guard's `.xml` clause is why an uncompressed intermediate is also checked. diff --git a/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-03-meanings-in-the-database.md b/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-03-meanings-in-the-database.md deleted file mode 100644 index ea72629..0000000 --- a/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-03-meanings-in-the-database.md +++ /dev/null @@ -1,114 +0,0 @@ ---- -phase: 3 -title: "Phase 3: Meanings in the database" -status: completed -priority: P1 -effort: "4h" -dependencies: [1] ---- - -# Phase 3: Meanings in the database - -## Overview - -Persist the senses phase 1 extracts in a `meanings` table, load them in `dictionary.Store` -behind a `Meanings(word)` method, and let the fixture word list carry meanings so tests and -the e2e suite have some. - -## Requirements - -- Functional: schema gains - ```sql - CREATE TABLE meanings ( - word TEXT NOT NULL, - ord INTEGER NOT NULL, - pos TEXT NOT NULL, -- Vietnamese part-of-speech label, '' when the heading was unmapped - gloss TEXT NOT NULL, - PRIMARY KEY (word, ord) - ) WITHOUT ROWID; - ``` - `ord` is the 0-based sense order from the page. No foreign key pragma; `verify()` checks - the join instead, as it does for aliases. (Validation session 1: `pos` column.) -- Functional: `finish()` takes the meanings map beside `words`; `writeTo` inserts them in the - same transaction. `meta` gains `meaning_count` (rows) and `words_with_meaning`. -- Functional: `verify()` adds: no meaning row whose word is missing from `words`; no empty - gloss; no gloss over 200 runes; `words_with_meaning` is at least 60% of `word_count` for a - `--dump` build (fixture builds are exempt — the check is passed a flag, or keyed on the - source table prefix). -- Functional: `dictionary.Store` loads `meanings` into `map[string][]Sense` ordered by - `ord`, where `type Sense struct { Pos, Gloss string }` is exported from the `dictionary` - package, and exposes `Meanings(word string) []Sense` returning nil for a word with none. - `Resolve` first: the caller passes a canonical word, as it does for `FirstSyllable`. - Startup log gains the meanings count next to `words`. -- Functional: `--words` accepts an optional meaning column: `word<TAB>sense<TAB>sense…`, - where a sense is `pos|gloss` or just `gloss` (the pipe never survives stripping, so it is a - safe separator). Lines without a tab have no meaning. `testdata/fixture-words.txt` gains hand-written - meanings for the words the e2e suite plays (phase 5 says which) and for at least half the - list, so the fixture database exercises the same store paths as the real one. -- Non-functional: `Store` memory grows by the text size, a few MB. Startup time unchanged - in practice (one more ordered scan). -- Non-functional: `builder_version = "5"` (set in phase 1; this is the contract it names). - -## Architecture - -``` -build-dictionary dictionary.Store - words map[string]entry ─┐ words map[string]wordInfo - meanings map[string][]sense ─┼─► sqlite ─► meanings map[string][]Sense - aliases map[string]string ─┘ Meanings(word) []Sense -``` - -The store's `validate()` already cross-checks `word_count`; add the same for -`meaning_count` so a database whose meanings table was truncated on disk is refused at -startup rather than served silently without meanings. - -## Related Code Files - -- Modify: `server/cmd/build-dictionary/main.go` — `finish`, `write`, `writeTo`, `verify`, - `runFromWordList` (tab-separated senses), `sourceSpec` unchanged -- Modify: `server/cmd/build-dictionary/main_test.go` — meanings written and verified; a - fixture list with tabs; a `verify` failure on an orphan meaning row -- Modify: `server/internal/dictionary/store.go` — `meanings` field, `loadMeanings`, - `Meanings()`, `validate` count check, `MeaningCount()` -- Modify: `server/internal/dictionary/store_test.go` — hand-built databases gain the table; - `Meanings` returns ordered senses and nil; a mismatched `meaning_count` is refused -- Modify: `server/cmd/noitu-server/main.go` — startup log line -- Modify: `testdata/fixture-words.txt` — meanings column; header comment documents the format -- Modify: `web/e2e/fixture-dictionary.js` — it reads the list one word per line - (`fixture-dictionary.js:15-19`); cut each line at the first tab before trimming, or the - e2e graph gains words that are really `word<TAB>sense` strings (validation session 1) - -## Implementation Steps - -1. Schema, insert, meta and `verify` in the builder; tests first for the orphan-row and - empty-gloss failures. -2. `runFromWordList` tab parsing; a test that a line with two tabs yields two senses in order, - `danh từ|…` splits into label and gloss, a cell without a pipe has an empty label, and a - line without a tab yields none. Update `fixture-dictionary.js` in the same step. -3. Store: load, method, validate; tests. -4. Fixture list: add meanings. Rebuild `data/fixture.db` (`make fixture-dict`) and start the - server against it; the log shows the count. -5. Rebuild the real database from phase 1's dump and confirm `verify` passes the 60% rule. - -## Success Criteria - -- [x] `go test ./cmd/build-dictionary ./internal/dictionary -race` green. -- [x] `sqlite3 data/noitu.db 'SELECT COUNT(*) FROM meanings'` is above 40,000 and - `words_with_meaning` above 60% of words (phase 6 records the actual). -- [x] `store.Meanings("học sinh")` on the real database returns a non-empty ordered slice - whose first sense has `Pos == "danh từ"`; on a word with no `#` line it returns nil. -- [x] `npm run test:e2e` still builds its word graph from the fixture list correctly (no - tab-carrying "words"). -- [x] The fixture database has meanings for the e2e words and the server starts on it. - -## Risk Assessment - -**The 60% rule is a guess.** Research counted 5,764 of 43,013 pages with no POS marker, and -some of those have no `#` line either; the true coverage is unknown until phase 1 runs. -Signal: `verify` fails on the real dump with a coverage in the 50s. Response: measure, set -the floor a comfortable margin under the measurement, and record why in the code comment. -The rule exists to catch a stripper that suddenly returns nothing, not to demand quality. - -**Fixture meanings drift from fixture words.** A word renamed in the list loses its meaning -silently. Signal: an e2e assertion on a meaning fails. Response: the builder's fixture path -logs words without meaning; keep the list short and hand-checked. diff --git a/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-04-meanings-on-the-wire.md b/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-04-meanings-on-the-wire.md deleted file mode 100644 index 79224da..0000000 --- a/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-04-meanings-on-the-wire.md +++ /dev/null @@ -1,108 +0,0 @@ ---- -phase: 4 -title: "Phase 4: Meanings on the wire" -status: completed -priority: P1 -effort: "3h" -dependencies: [3] ---- - -# Phase 4: Meanings on the wire - -## Overview - -Carry a word's senses in the two messages that introduce a word to the client — -`PlayedWord` and `GameStarted` — regenerate the Go and JS types and the binary fixtures, -and have the room attach the senses where it already renders those messages. - -## Requirements - -- Functional: `proto/noitu/v1/game.proto` gains - ```proto - // One definition of a word as Wiktionary gives it: the part of speech it - // sits under, in Vietnamese ("danh từ"), empty when the heading was not one - // the builder knows, and the definition stripped of markup at at most 200 - // characters. Plain text both: the client renders them as text, never as - // markup. - message Sense { - string pos = 1; - string gloss = 2; - } - message PlayedWord { - … - // At most five. Empty when the dictionary has no definition for the word. - repeated Sense meanings = 7; - } - message GameStarted { - … - repeated Sense opening_meanings = 9; - } - ``` - (Validation session 1: `Sense` instead of a bare string, so the label is a field.) -- Functional: `wsapi.Dictionary` gains `Meanings(word string) []dictionary.Sense`. - `game.Dictionary` does not change; the engine never reads a meaning. `convert.go` gains - `Senses([]dictionary.Sense) []*noituv1.Sense`. -- Functional: `convert.PlayedWord` takes the senses (`PlayedWord(m game.Move, byMe bool, - meanings []dictionary.Sense)`) and its one caller, `sendTurnUpdate` (`room.go:903`, also - used by the resume path), passes `r.dict.Meanings(move.Word)`; `sendGameStarted` fills - `OpeningMeanings` from `r.dict.Meanings(r.opening)`. Both paths are per recipient already; - the lookup happens once per move, before the per-seat loop. -- Functional: the reconnect path (`GameStarted` + last `TurnUpdate`) carries meanings - without further change; a test asserts a resumed client sees the opening's and last word's - senses. -- Functional: `buf generate` regenerates `server/gen/` and `web/src/lib/proto/`; - `go test ./internal/wsapi -update` regenerates `proto/testdata/`; the JS decode test reads - the new field. -- Functional: the hand-built dictionaries in `wsapi` tests implement `Meanings`; at least - one returns senses so the assertion is not vacuous. -- Non-functional: frame size grows by up to ~1 KB per move. Nothing in the client or server - caps frames near that. - -## Architecture - -``` -room.handleSubmit ── engine.Submit ──► Move ──► meanings := r.dict.Meanings(move.Word) - for each seat: PlayedWord(move, byMe, meanings) -room.sendGameStarted ─────────────────────────► OpeningMeanings: r.dict.Meanings(r.opening) -``` - -The bot room path renders through the same `sendTurnUpdate`, so bot moves carry meanings -without a second change. - -## Related Code Files - -- Modify: `proto/noitu/v1/game.proto` -- Regenerate: `server/gen/noitu/v1/game.pb.go`, `web/src/lib/proto/**`, `proto/testdata/*` -- Modify: `server/internal/wsapi/room.go` — `Dictionary` interface, `sendGameStarted`, - the `PlayedWord(` call in `sendTurnUpdate` -- Modify: `server/internal/wsapi/convert.go` — `PlayedWord` signature, `Senses` -- Modify: `server/internal/wsapi/*_test.go` — fake dictionaries gain `Meanings`; new - assertions on `Played.Meanings` and `OpeningMeanings`, including after a resume -- Modify: `web/tests/game-wire.test.js` — the fixture decode asserts the new fields - -## Implementation Steps - -1. Edit the schema; `buf lint`; `buf generate`. -2. Interface and room changes; fix compile errors in tests by adding `Meanings` to fakes. -3. Add the assertions; `go test ./internal/wsapi -update` then `go test ./... -race`. -4. `cd web && npm test` — the wire test decodes the regenerated fixtures. -5. `make proto-check` — committed generated code matches the schema. - -## Success Criteria - -- [x] `buf lint` clean; `make proto-check` clean. -- [x] A bot game and a room game deliver `Played.Meanings` with `Pos` and `Gloss` for a word - the fake dictionary has senses for, and an empty list for one it does not. -- [x] A resumed session's `GameStarted.OpeningMeanings` and replayed `TurnUpdate.Played.Meanings` - are populated. -- [x] `proto/testdata/` fixtures regenerated and the JS suite decodes them. - -## Risk Assessment - -**Field numbers.** `PlayedWord` uses 1–6, `GameStarted` 1–8; 7 and 9 are free and nothing -is reserved in either. Signal: `buf breaking` complaint. Response: none expected; adding a -repeated field is wire-compatible, and there is one client. - -**Meanings on `PlayerEliminated.suggestions`.** Out of scope by the plan's non-goals; a -reviewer may ask. Response: the suggestions are shown to somebody who just lost the turn, -a list of words, and a meaning under each would be a second UI. Not this plan. diff --git a/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-05-meanings-in-the-client.md b/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-05-meanings-in-the-client.md deleted file mode 100644 index 9629ce4..0000000 --- a/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-05-meanings-in-the-client.md +++ /dev/null @@ -1,133 +0,0 @@ ---- -phase: 5 -title: "Phase 5: Meanings in the client" -status: completed -priority: P1 -effort: "5h" -dependencies: [4] ---- - -# Phase 5: Meanings in the client - -## Overview - -Show a word's meanings under it in the chain, open for the newest word, closed for the rest, -with a click on any word toggling its own; keep the open/closed set in the store so the -rules are testable without a browser, and cover the behaviour end to end. - -## Requirements - -- Functional: `ChainEntry` gains `meanings: {pos: string, gloss: string}[]`, filled from - `played.meanings` and `value.openingMeanings`. -- Functional: the store keeps `expanded: Set<string>` of words whose meaning is open, and - `toggleMeaning(word)`. Any number of words may be open at once. Rules (validation - session 1: they apply to every word, with or without definitions): - - `gameStarted`: `expanded = {openingWord}`. - - `turnUpdate` with a word: remove the previous newest word (`chain[chain.length-1].word` - before the push), add the new word. A `turnUpdate` without a word (an elimination) - changes nothing. - - `toggleMeaning(word)`: flip membership. - - `reset()` clears the set. A resumed session gets `gameStarted` then one `turnUpdate` and - ends up with only the last word open, which is the same as a fresh one. -- Functional: `ChainHistory.svelte` renders every word as a `<button type="button">` with - `aria-expanded` and `aria-controls` pointing at its panel. The panel is shown when the word - is in `expanded`: an `<ol class="meanings">` with one `<li>` per sense rendered as - `(pos) gloss`, or `gloss` alone when `pos` is empty; when the word has no senses the panel - is a single `<p class="meanings none">` with `t.meaningNone`. Everything else in the row — - who played it, badges, points, the correction note — is unchanged. -- Functional: strings in `web/src/lib/i18n/vi.js`: `meaningShow` / `meaningHide` for the - button's `aria-label` (`Xem nghĩa của {word}` / `Ẩn nghĩa của {word}`) and `meaningNone` - (`Chưa có nghĩa`). -- Functional: the auto-scroll effect keeps working — the newest row is at the top and its - open list is what the scroll lands on. -- Functional: `history-export.js` is unchanged (non-goal). -- Non-functional: the meanings are rendered as text (`{sense}`), never with `{@html}`. -- Non-functional: the row stays legible on a phone: the list wraps under the word at full - row width (`flex-basis: 100%`), muted colour, ~0.85rem, numbered by the `<ol>`. -- Non-functional: `prefers-reduced-motion` respected if any open/close transition is added; - the simplest is none. - -## Architecture - -``` -game.svelte.js - state.chain[i].meanings from the wire - state.expanded Set<string>, client-only - toggleMeaning(word) - -ChainHistory.svelte - <li> - <button class="word" type="button" aria-expanded={open} aria-controls={id} - aria-label={fill(open ? t.meaningHide : t.meaningShow, { word: entry.word })} - onclick={() => game.toggleMeaning(entry.word)}>{entry.word}</button> - … by / meta / corrected as today … - {#if open} - {#if entry.meanings.length} - <ol class="meanings" {id}> - {#each entry.meanings as sense}<li>{sense.pos ? `(${sense.pos}) ` : ''}{sense.gloss}</li>{/each} - </ol> - {:else} - <p class="meanings none" {id}>{t.meaningNone}</p> - {/if} - {/if} - </li> -``` - -`id` is derived from the row's index in the chain, not from the word, so two rows never -share one (a word is never played twice, but the opening word could in principle be typed -as a later variant). - -The e2e helper `chainWords` selects `ol li .word`; the class stays on the button so the -helper and every existing spec keep working, and the nested `<ol class="meanings">` has no -`.word` inside it. - -## Related Code Files - -- Modify: `web/src/lib/stores/game.svelte.js` — `ChainEntry` typedef, `expanded`, - `toggleMeaning`, the `gameStarted`/`turnUpdate`/`reset` branches -- Modify: `web/src/lib/components/ChainHistory.svelte` — markup and styles -- Modify: `web/src/lib/i18n/vi.js` — three strings -- Modify: `web/tests/game-store.test.js` — the expansion rules above, each as a case -- Modify: `web/tests/i18n.test.js` — only if it enumerates keys -- Modify: `web/e2e/helpers.js` — `openMeanings(page)` returning the words whose list is - visible; `web/e2e/bot-game.spec.js` — newest open, previous closed after a move, click - toggles; `web/e2e/pvp-game.spec.js` — the other browser sees the same word open -- Modify: `testdata/fixture-words.txt` — meanings for the words the specs play (phase 3) - -## Implementation Steps - -1. Store first, with tests: opening open; new word closes previous and opens itself; a word - with no meanings is opened the same way; elimination leaves the set alone; toggle flips - and two words can be open together; reset clears; the resume sequence ends with one open. -2. Component markup and styles; check both themes and a 360px viewport. -3. i18n strings; run the copy test. -4. e2e: fixture meanings, helper, three assertions. CI runs the browser; locally, run what - Playwright allows. -5. `npm run check && npm test`. - -## Success Criteria - -- [x] `npm run check && npm test` green; store tests cover all seven rules. -- [x] In a bot game the opening word's meaning is open; after the bot's first move only the - bot's word is open; clicking the opening word opens it again while the bot's stays - open, and clicking the bot's word closes it. -- [x] Senses render as `(danh từ) …`; a sense with an empty label renders the gloss alone; a - word without meanings opens to `Chưa có nghĩa`. -- [x] e2e specs assert the three behaviours and pass in CI. -- [x] Nothing in the transcript export changed. - -## Risk Assessment - -**The button steals focus from the word input.** The README says focus is left alone while -a player types, and a chain row that grabs focus on render would break that. Signal: typing -interrupted after a move. Response: never call `focus()` in the component; the button is -only focusable by the user's own click or Tab. - -**Long meanings push the chain off screen on a phone.** Five senses of 200 characters is a -tall row. Signal: the newest row fills the viewport. Response: the cap is server-side and -already chosen; if it proves too tall, `max-height` with `overflow-y: auto` on the list is -a style change. Do not shorten text client-side — it would differ from what the server sent. - -**A resumed session opens two words.** `gameStarted` opens the opening word; the replayed -`turnUpdate` must close it. Covered by the rule ordering and a store test using the resume -sequence. diff --git a/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-06-measure-and-attribute.md b/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-06-measure-and-attribute.md deleted file mode 100644 index e4d6e80..0000000 --- a/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-06-measure-and-attribute.md +++ /dev/null @@ -1,122 +0,0 @@ ---- -phase: 6 -title: "Phase 6: Measure and attribute" -status: completed -priority: P1 -effort: "3h" -dependencies: [2, 3] ---- - -# Phase 6: Measure and attribute - -## Overview - -Record what the first dump-built database contains against the kaikki one, sample the -stripper's output, and rewrite every attribution surface so it names the Wikimedia dump, -drops kaikki.org and wiktextract, and stops claiming the database holds only word forms. - -## Requirements - -- Functional: `measurement.md` beside this plan, same shape as the kaikki plan's: the file - measured (URL, SHA-256, `source_pages`, `source_fetched_at`, size), the build log, the - graph metrics (words, syllables, openers, openers ≥2, dead ends) old vs new, bot-vs-bot at - 60 games, the corpus diff (shared / gained / lost with samples), wall time of the build, - and the meanings numbers: `meaning_count`, `words_with_meaning` and its percentage, the - distribution of senses per word, the share of senses with an empty part-of-speech label - and the unmapped heading codes by frequency, the ten commonest dropped template names, and - a 50-sense random sample read by a person with each marked fine / terse / wrong. -- Functional: `data/ATTRIBUTION.md` — source table names the dump - (`viwiktionary-latest-pages-articles.xml.bz2`, monthly, unpinned), removes the wiktextract - row and the LREC citation, states the chain is now Wiktionary tiếng Việt contributors → - this project. Modifications list: item 1 becomes "section selection — the Vietnamese - section of each page, either markup dialect; redirects skipped"; a new item describes the - definition text: taken from `#` lines, markup stripped, templates other than the listed - ones removed, capped at five senses of 200 characters, each labelled with the Vietnamese - name of the part-of-speech heading it sat under, so the text is an *excerpt and - modification* of the entry, not the entry; item 8 ("dropped fields") says everything else - — examples, translations, pronunciations, etymologies, categories — is dropped, and the - sentence "contains only word forms, not meanings" is deleted. The - reproduce section names the new commands. -- Functional: `NOTICE` — affected artifacts name the dump file; the extraction lines go; - "identified by the SHA-256 recorded in the database's meta table" stays. -- Functional: README — licence section drops wiktextract/kaikki; Setup and the raw commands - already changed in phase 2; a sentence under the game description says the chain shows - each word's Wiktionary meaning. -- Functional: `docs/deployment.md` "Updating the dictionary" says dumps.wikimedia.org and - monthly. -- Functional: the in-game footer (`AttributionFooter.svelte`, `attributionSource`) already - says Wiktionary tiếng Việt and links vi.wiktionary.org; verify and leave. -- Functional: `verify` in the builder and the pin test are the only two places allowed to - encode numbers from this measurement (the coverage floor; the URL). Nothing else. -- Non-functional: the measurement is a record, not a gate — the plan's decision that nothing - regresses is checked against the numbers, and a regression is reported to the owner, not - silently accepted or silently fixed. - -## Architecture - -```sh -cp data/noitu.db /tmp/kaikki.db # before the first dump build -make fetch-dict && make dict -cd server && go test ./internal/bot/ -run RealCorpus -v -count=1 # per database -``` - -```sql -SELECT COUNT(*) FROM words; SELECT COUNT(*) FROM syllables; -SELECT COUNT(*) FROM (SELECT first FROM words GROUP BY first HAVING COUNT(*) >= 2); -SELECT COUNT(*) FROM syllables WHERE out_degree = 0; -SELECT COUNT(*) FROM meanings; SELECT COUNT(DISTINCT word) FROM meanings; -SELECT pos, COUNT(*) FROM meanings GROUP BY pos ORDER BY 2 DESC; -- '' is the unmapped share -SELECT n, COUNT(*) FROM (SELECT word, COUNT(*) n FROM meanings GROUP BY word) GROUP BY n; -SELECT word, gloss FROM meanings ORDER BY RANDOM() LIMIT 50; -ATTACH '/tmp/kaikki.db' AS old; -SELECT COUNT(*) FROM words w JOIN old.words o ON o.word = w.word; -- shared -SELECT word FROM words WHERE word NOT IN (SELECT word FROM old.words) ORDER BY RANDOM() LIMIT 25; -- gained -SELECT word FROM old.words WHERE word NOT IN (SELECT word FROM words) ORDER BY RANDOM() LIMIT 25; -- lost -``` - -## Related Code Files - -- Create: `plans/260908-2056-wiktionary-dump-corpus-and-meanings/measurement.md` -- Modify: `data/ATTRIBUTION.md`, `NOTICE`, `README.md`, `docs/deployment.md` -- Verify only: `web/src/lib/components/AttributionFooter.svelte`, `web/src/lib/i18n/vi.js` - -## Implementation Steps - -1. Before the first real dump build, copy today's `data/noitu.db` aside. -2. Build; run the queries and the real-corpus bot tests on both databases; fill - `measurement.md`. -3. Read the 50 senses; mark each; if more than ~10 are wrong for the same reason, that is a - phase-1 stripper fix, made now, and the sample re-drawn. -4. Rewrite the four documents. Run `npx vitest run tests/dictionary-source.test.js` (README - and ATTRIBUTION quote the URL). -5. `grep -rniI "kaikki\|wiktextract\|jsonl\|only word forms" --exclude-dir=plans .` finds - nothing. -6. Plan status via `ak plan`; journal. - -## Success Criteria - -- [x] `measurement.md` complete with every table above; words ≥ 34,813 and no graph metric - below the kaikki database's, or the regression is written up and shown to the owner. -- [x] Meaning coverage recorded; the 50-sense sample recorded with verdicts. -- [x] `data/ATTRIBUTION.md`, `NOTICE`, README and `docs/deployment.md` describe the dump and - the definition text accurately; the pin test passes. -- [x] The grep in step 5 is empty. - -## Risk Assessment - -**The dump has fewer words than kaikki for some syllables.** Research measured the dump as a -near-superset (36,200 vs 34,813 through the same filter), but the two are different months -by the time this runs. Signal: `lost` is in the thousands. Response: list them, look for a -parser cause (a section boundary cut, a dialect miss) before accepting; a genuine wiki -deletion is accepted and recorded. - -**The stripper sample is bad.** Signal: more than a fifth of 50 marked wrong. Response: fix -the stripper for the commonest cause in this plan; ship with the rest recorded; the -follow-up is a plan of its own, because it is template-by-template work against a moving -wiki. - -**Attribution says less than the law wants.** CC BY-SA 4.0 asks for attribution, a licence -link, and an indication of modifications. The definition text makes the "modifications" -line load-bearing: the database now redistributes edited excerpts of the entries. The -ATTRIBUTION rewrite above is the compliance; a reviewer should read it against the licence -text once. Signal: none in code. Response: do the review. diff --git a/plans/260908-2056-wiktionary-dump-corpus-and-meanings/plan.md b/plans/260908-2056-wiktionary-dump-corpus-and-meanings/plan.md deleted file mode 100644 index f51093e..0000000 --- a/plans/260908-2056-wiktionary-dump-corpus-and-meanings/plan.md +++ /dev/null @@ -1,283 +0,0 @@ ---- -title: "Wiktionary dump corpus and word meanings" -description: "Build data/noitu.db from the Wikimedia dump of Wiktionary tiếng Việt instead of kaikki.org's export, carry each word's definitions into the database, and show them in the chain: click to toggle, newest word open by default" -status: completed -priority: P1 -effort: "~2.5d" -tags: [dictionary, data, proto, web, licensing, build] -created: 2026-09-08 -blockedBy: [] -blocks: [] ---- - -# Wiktionary dump corpus and word meanings - -## Overview - -`data/noitu.db` is derived from kaikki.org's wiktextract export of Wiktionary tiếng Việt -(plan `260908-1653-kaikki-viwiktionary-corpus`, completed today): 34,813 words, word forms -only, fetched unpinned. Two things change here. - -**The source becomes the Wikimedia dump itself.** `dumps.wikimedia.org/viwiktionary/` is the -file kaikki extracts from. Reading it directly removes the intermediary, its weekly refresh -and its `vi` extractor, which today loses ~1,400 real words the dump has (measured in -`plans/reports/research-260908-1529-viwiktionary-dump-measured.md`: **36,200** accepted -titles against kaikki's 34,813, with reduplicatives such as `rưng rức`, `nháo nhác`, `óc ách` -among the difference). The cost is a wikitext parser of our own — both markup dialects the -wiki currently mixes — instead of a JSON decoder. - -**The database gains meanings.** Each word carries its Wiktionary definitions, stripped of -markup, capped at five senses of 200 characters. The client shows them under the word in the -chain: the newest word's meaning is open by default, a new word closes the previous one and -opens its own, and a click on any word toggles its meaning. This is the reason the source -matters: kaikki's glosses are pre-cleaned, but the dump's definitions cover the words kaikki -drops, and a stripper for `# ...` lines is a bounded job. - -Owner decisions taken before this plan was written: - -- **Track `latest/`, unpinned.** Same posture as today: `curl` whatever - `viwiktionary-latest-pages-articles.xml.bz2` currently is, record its SHA-256 in `meta`. - Dated directories exist and would pin honestly; the owner chose freshness. Wikimedia's dump - is monthly, so "fresh" now means monthly rather than weekly. -- **All senses, capped, each with its part of speech.** Every definition line of the - Vietnamese section travels, cut at 5 senses and 200 characters each, and each sense carries - the Vietnamese label of the heading it sat under (`danh từ`, `động từ`, `tính từ`, `danh từ - riêng` …), shown as a prefix: `(danh từ) Trẻ em học tập ở nhà trường.` A heading the - builder cannot map yields an empty label, never a dropped sense. -- **Every word is clickable.** A word with no definition still opens, to one line saying - `Chưa có nghĩa`, so the chain behaves the same for every word. -- **Nothing is dropped by label.** Carries over from the previous two plans: part of speech - and capitalization are tallied in the build log and never filter. `Danh từ riêng` - (`pr-noun`) labels are counted, not acted on. -- **One corpus input mode.** `--kaikki` is replaced by `--dump`, not kept beside it. `--words` - stays for the fixture and gains an optional meaning column so the e2e suite can see one. - -## What changes, in numbers - -| | today (kaikki) | after (dump) | -|---|---|---| -| words | 34,813 | **~36,200** (measured on the 2026-09-01 dump; phase 6 records the actual) | -| meanings | none | one to five per word; coverage measured in phase 6 (~5,700 pages have no `#` line and will have none) | -| source | 62 MB JSONL, weekly, unpinned | **~61 MB `.xml.bz2`, monthly, unpinned** | -| parser | `encoding/json`, 3 fields | `compress/bzip2` + `encoding/xml` + a wikitext section/definition scanner | -| `--min-words` floor | 30,000 | 30,000 (unchanged: 36,200 measured, and a dialect the parser misses is ~7,000 pages) | -| data licence | CC BY-SA 4.0 | CC BY-SA 4.0 (the database now carries definition *text*, so the share-alike obligation is more literal, not different) | -| wire | `PlayedWord` 6 fields | `message Sense {pos, gloss}`; `PlayedWord.meanings`, `GameStarted.opening_meanings` | -| `builder_version` | 4 | 5 | - -## The source - -``` -URL https://dumps.wikimedia.org/viwiktionary/latest/viwiktionary-latest-pages-articles.xml.bz2 -Dated https://dumps.wikimedia.org/viwiktionary/20260901/viwiktionary-20260901-pages-articles.xml.bz2 - 63,513,513 bytes, MD5 6c2491e703e7d946f23a405996b4d172 (published), monthly on the 1st -Shape <mediawiki><page><title/><ns/><id/>[<redirect title=""/>]<revision><text>wikitext</text></revision></page>… -``` - -391,543 pages; 349,461 in namespace 0; **43,013 with a Vietnamese section**; 3,237 of those -are redirects (834 case-only, `mặt trời` → `Mặt Trời`) and are skipped, because the target -page is read on its own and lowercased by `accept()`. - -Two markup dialects are live and the mix shifts monthly: - -| | legacy (35,885 pages) | new (7,128 pages) | -|---|---|---| -| Vietnamese section opens | `{{-vie-}}` | `== {{langname\|vi}} ==` | -| section closes | next `{{-xxx-}}` whose code is *not* a known section code (`-eng-`, `-fra-`, `-tyz-` …) | next level-2 heading `== … ==` | -| POS heading | `{{-noun-}}`, `{{-verb-}}`, `{{-pr-noun-}}`, `{{-place-}}` … | `=== {{ĐM\|noun}} ===`, `{{vi-noun}}`, `{{vi-pr-noun}}` … | -| definition | a line starting `# ` (not `#:`, `#*`) under a POS heading | same | - -Sample definitions as they sit in the dump and as they must come out: - -``` -{{-noun-}} … # [[chỗ|Chỗ]] [[râm]] [[mát]], do [[trời]] có [[mây]] hoặc do không bị [[nắng]] [[chiếu]]. -→ pos "danh từ" gloss "Chỗ râm mát, do trời có mây hoặc do không bị nắng chiếu." - -=== {{ĐM|pr-noun}} === … # {{place|vi|thủ đô|c/Việt Nam}}. -→ pos "danh từ riêng" gloss "thủ đô, Việt Nam." - -# {{label|vi|thuộc lịch sử}} Một [[tỉnh]] cũ của [[Việt Nam]] vào nửa cuối thế kỷ XIX. -→ pos "danh từ riêng" gloss "(thuộc lịch sử) Một tỉnh cũ của Việt Nam vào nửa cuối thế kỷ XIX." -``` - -The client renders a sense as `(pos) gloss`, or `gloss` alone when the label is empty. - -## Decisions taken - -- **Parse in Go, inside `build-dictionary`.** `compress/bzip2` and `encoding/xml` are stdlib; - the Docker `dict` stage and the Makefile keep one command. Go's bzip2 is pure Go and slow - (tens of MB/s); a ~400 MB decompressed stream is well under a minute. If phase 1 measures - more than two minutes, the fallback is `bzip2 -dc` in the Makefile feeding a plain `.xml` - — a flag change, recorded as a risk below, not a redesign. -- **The stripper is small and lossy on purpose.** Links keep their display text; bold and - italic markers go; `<ref>` and comments go; `{{label|vi|x}}`/`{{lb|vi|x}}`/`{{gloss|x}}` - become `(x)`; `{{l|vi|x}}`, `{{vi-l|x}}`, `{{w|x}}` become `x`; `{{place|vi|a|b}}` keeps its - positional parameters after the language code with any `c/` prefix removed; **every other - template is dropped whole**, nesting handled by a depth counter. What survives is trimmed - and whitespace-collapsed; an empty result is not a sense. This will leave some definitions - terse or odd. Phase 6 samples 50 and records what the stripper got wrong; fixing template - by template is a follow-up, not this plan. -- **Meanings live in their own table** — `meanings(word, ord, pos, gloss)` — and are loaded - into memory by the store with the words. The label is a column, not baked into the gloss: - the 200-character cap applies to the definition, and the client decides how a label is - shown. Ten kilobytes of text per hundred words is a few - megabytes for the corpus; a per-move query would be a second code path for nothing. -- **The engine never sees a meaning.** `game.Dictionary` is unchanged. `wsapi.Dictionary` - gains `Meanings(word) []dictionary.Sense` and the room attaches them when it renders a `PlayedWord` - or a `GameStarted`, the same place `by_me` is decided. A reconnect already replays only the - opening and the last move, and both carry meanings, so nothing new is needed there. -- **Expansion state is client-only.** Which words are open is UI state, like the theme. The - store keeps a set of open words — any number may be open at once; `gameStarted` opens the - opening word, a `turnUpdate` with a word closes the previous newest and opens the new one, - a click toggles. These rules apply to every word, with or without a definition. The - transcript export is unchanged. -- **Meaning text is plain text end to end.** It is rendered as text, never as HTML; the - builder strips markup, the client escapes as Svelte does by default. Wiki text is - user-generated content and travels through the same sanitizer path as any other string - the server has not written itself: the builder caps length, drops control characters and - collapses whitespace so nothing the store loads can be shaped like a chat injection. - -## Phases - -| # | Phase | Status | -|---|-------|--------| -| 1 | [Read the dump](./phase-01-start.md) | Completed | -| 2 | [Fetch the dump](./phase-02-fetch-the-dump.md) | Completed | -| 3 | [Meanings in the database](./phase-03-meanings-in-the-database.md) | Completed | -| 4 | [Meanings on the wire](./phase-04-meanings-on-the-wire.md) | Completed | -| 5 | [Meanings in the client](./phase-05-meanings-in-the-client.md) | Completed | -| 6 | [Measure and attribute](./phase-06-measure-and-attribute.md) | Completed | - -Phases 1 and 3 land together: a reader that extracts definitions and a schema that has -nowhere to put them is half a change. Phase 2 can land with them or just after. Phases 4 -and 5 are one wire change and its consumer; the schema regenerates once. Phase 6 is -measurement and the attribution rewrite, and the attribution must land in the same commit -as the first dump-built database, because `data/ATTRIBUTION.md` today says the database -"contains only word forms, not meanings", which stops being true. - -## Non-goals - -- Dropping words by proper-noun label, capitalization or part of speech. -- Keeping kaikki as a fallback or second source. -- Meanings for the suggestions shown to an eliminated player, in the transcript export, or - anywhere outside the chain. -- A dictionary lookup UI. The meaning is shown for words that were played. -- Pinning the dump. Recorded as the open question it already was. - -## Success criteria - -- [x] `make fetch-dict && make dict` downloads `viwiktionary-latest-pages-articles.xml.bz2` - and builds `data/noitu.db` with more than 30,000 words and a meaning for the large - majority of them; the build log reports pages seen, pages with a Vietnamese section, - pages per dialect, redirects skipped, POS tallies and definition counts. -- [x] A truncated or non-bzip2 download, an XML stream that ends mid-page, and a page count - far below the norm each fail the build with a message naming the cause. -- [x] `meta` records `source_url`, `source_sha256`, `source_pages`, `source_fetched_at`, - `meaning_count` and `builder_version = 5`; `source_rows` is gone. -- [x] `Sense`, `PlayedWord.meanings` and `GameStarted.opening_meanings` are in the schema, - the generated Go and JS trees, and the binary fixtures under `proto/testdata/`. -- [x] In a bot game and a two-browser online game the newest word's meaning is open, the - previous word's closes when a new one arrives, and clicking any word toggles its own. - Senses show as `(danh từ) …`; a word with no definition opens to `Chưa có nghĩa`. -- [x] `data/ATTRIBUTION.md`, `NOTICE`, README, `docs/deployment.md`, the Dockerfile and the - builder's constants name the Wikimedia dump and no longer name kaikki.org or - wiktextract; the "only word forms" claim is replaced by an accurate description of the - definition text carried. -- [x] `go vet ./... && go test ./... -race`, `npm run check && npm test`, `npm run test:e2e` - (CI) and both Docker image variants are green; the CI leak guard rejects a `.bz2` or - `.xml` inside the image. -- [x] Phase 6's measurement is recorded beside this plan: word count, meaning coverage, - graph metrics and bot game lengths against the kaikki database, plus the 50-sense - stripper sample. - -## Open questions - -- **Unpinned `latest/`.** Wikimedia repoints `latest` once a month; a dump run can also fail - partway and leave `latest` on the previous month, which is harmless. Accepted by the owner, - same as for kaikki. `source_sha256` and `source_fetched_at` (the file's server-side - modification time, kept by `curl -R`) identify a build. Should reproducibility ever be - wanted, the dated URL is a one-variable change. -- **Definitions that are only a template.** `# {{place|vi|thủ đô|c/Việt Nam}}.` strips to - `thủ đô, Việt Nam.` — readable, not prose. Definitions that are *only* an unknown template - strip to nothing and the sense is dropped; if that leaves a word with no sense, the word - has no meaning shown. Phase 6 counts how many and lists the commonest template names so - the next round knows which to teach the stripper. -- **Headings the label map does not know.** `{{-xxx-}}` and `{{ĐM|xxx}}` codes outside the - map give an empty label; the sense is kept. Phase 6 lists unmapped codes by frequency so - the map can grow; a code seen on more than a few hundred definitions is added in phase 1. -- **Dialect drift.** The wiki is migrating legacy `{{-vie-}}` pages to `== {{langname|vi}} ==`. - Both are parsed; the build log prints the count per dialect so a month where one falls to - zero is visible. A third form appearing would show as a drop in `source_pages` and, - eventually, the floor. - -## Validation Log - -### Session 1 — 2026-09-08 - -Verification (Full tier, 6 phases; claims checked by reading the code while the plan was -written, then re-grepped): 14 checked, 13 verified, 1 failed, 0 unverified. - -- FAILED → fixed: `web/e2e/fixture-dictionary.js` reads `testdata/fixture-words.txt` as one - word per line (`fixture-dictionary.js:15-19`); the tab-separated meaning column must be - stripped there. Phase 3 now lists it as a definite change, not a conditional one. -- Corrected: `PlayedWord(` has one caller, `room.go:903` inside `sendTurnUpdate`; the resume - path reuses it. Phase 4 no longer implies a second call site. -- Verified: `wsapi.Dictionary` at `room.go:268`, `sendGameStarted` at `room.go:744`, - resume replay at `room.go:1190-1195`, `Store.validate` word-count check at `store.go:213`, - `reset()` at `game.svelte.js:198`, `chainWords` selector `ol li .word` at - `e2e/helpers.js:184`, `TestDifficultyLadderRealCorpus` and `TestHardChooseLatencyRealCorpus` - in `bot/realcorpus_test.go`, `web/tests/game-wire.test.js` decodes `proto/testdata`, - `PlayedWord` fields 1–6 and `GameStarted` 1–8 with nothing reserved, `web/tests/i18n.test.js` - does not enumerate keys (phase 5's conditional line resolves to no change), Go 1.25. - -Questions asked: 4. - -| Question | Decision | Effect | -|---|---|---| -| Open state when another word is opened | Independent toggles; only the automatic close of the previous newest | Phase 5 rules unchanged; stated explicitly | -| Part-of-speech label on senses | **Prefix each sense with its POS** | `meanings.pos` column; `message Sense {pos, gloss}`; heading→label map in phase 1; client renders `(pos) gloss` | -| Meanings in the transcript download | No, chain only | Non-goal stands | -| A word with no definition | **Clickable, opens to `Chưa có nghĩa`** | Every word is a button; `meaningNone` string; auto-open applies to all words | - -### Whole-Plan Consistency Sweep - -Re-read `plan.md` and all six phase files after propagation. Searched for `no toggle`, -`no button`, `nothing to click`, `[]string`, `repeated string meanings`, `(word, ord, gloss)`, -`if it parses`, `wherever PlayedWord`. All occurrences reconciled to the decisions above. No -unresolved contradictions. - -### Session 2 — 2026-09-08, implementation - -All six phases implemented in one working tree; measurement in [`measurement.md`](./measurement.md). - -| Claim in the plan | Found | Effect | -|---|---|---| -| `pron` → đại từ in the label map | `pron` is *pronunciation* on this wiki (36,533 legacy + 5,501 new headings); pronoun is `pronoun`/`per-pronoun` | map corrected; `pron` is a non-POS section code | -| New dialect headings are `{{ĐM|code}}` | also `{{section|code}}` (3,506) and shorthand codes `n`/`v` | both forms and both shorthands mapped | -| `{{-dfn-}}` is a heading | it is a "definitions" marker placed under a POS heading | transparent to the label | -| Bot's bzip2 may take > 2 min | 21–25 s for the whole build | no fallback | -| Empty-label senses under 10% | 11.9%, all from pages with no POS heading at all; every code seen more than once is mapped | accepted, recorded | -| Fixture bzip2 lives at `testdata/mini-dump.xml.bz2` | placed in the package's own `server/cmd/build-dictionary/testdata/` (Go convention; the repo-root `testdata/` is the Makefile's and e2e's) | path only | -| `expanded: Set<string>` in the store | a `string[]` with set semantics, because Svelte 5's `$state` proxies arrays and not Sets | same rules, same tests | -| Stripper teaches only `label`/`l`/`place` families | also `nhãn`/`context`/`term` (label spellings), `n-g`, and `see-entry`/`like-entry` → `Xem x`, the commonest definition-line templates | phase-1 risk response, recorded | -| First build lost 15 kaikki words | pages with `{{-vie-}}{{-pron-}}…{{-place-}}` on one line | scanner reads several headings per line; second build lost 0 | - -Numbers: 36,200 words (34,813 shared with kaikki, 1,387 gained, 0 lost), 40,842 senses on -35,062 words (96.9%), 43,011 pages, no graph metric below the kaikki database's. - -Verification: `go vet ./... && go test ./... -race` green; `npm run check && npm test` green -(183); `npx playwright test` 38/39 on each full run, the one failure being the four-player -seating spec ("a fifth is turned away"), which no changed file touches and which fails about -one run in three on this machine with local Chrome **on `HEAD` as well** (stashed tree, -`--repeat-each 4`: 3 passed, 1 failed) — a pre-existing flake, not a regression; both meanings -specs pass every run; Docker image variants -**not built locally** — the daemon was not running — and left to CI. `--min-pages` (default -20,000) was added to the builder so the page floor is a flag like `--min-words`, which the -tests need to lower. - -Code review (`plans/reports/code-reviewer-260908-2210-…md`): no critical finding; one high -(a flaky e2e assertion on the opening's label), three medium (nested-list styling leak, -format characters surviving the stripper, opaque refusal of a v4 database) and eight low, all -fixed in the same tree and re-verified; see `measurement.md` "After code review". - -<!-- slug: wiktionary-dump-corpus-and-meanings -->