tiennm99/noitu
1
0 Fork 0
mirror of https://github.com/tiennm99/noitu.git synced 2026-10-14 02:16:25 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(plans): drop delivered plan phases, keep journals and reports

Browse Source
This commit is contained in:
tiennm99 committed 2026-09-10 09:56:31 +07:00
1 parent 2f893c9877
commit b1204e270f
42 files changed
-5977
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

plans/260904-1125-noi-tu-web-game/phase-01-foundations-and-data-pipeline.md
-146
View File
@@ -1,146 +0,0 @@
---
title: "Phase 1: Foundations and Data Pipeline"
status: done
phase: 1
priority: P1
effort: "3d"
dependencies: []
---
# Phase 1: Foundations and Data Pipeline
## Overview
Stand up the repo skeleton and produce `data/noitu.db` — the derived, normalized Vietnamese
wordlist of words with **at least 2 syllables** — from the upstream `minhqnd/dictionary`
SQLite release, with full CC BY-SA 4.0 compliance. Nothing downstream can be built or
tested without this data.
## Requirements
**Functional**
- [x] Reproducible build: upstream `dictionary.db` → `data/noitu.db`, runnable by any contributor
- [x] Output contains only Vietnamese (`lang_code = 'vi'`) entries of **2 or more space-separated syllables**
- [x] All words NFC-normalized and lowercased
- [x] Tone-placement aliases resolved (`hoà`→`hòa`, `thuý`→`thúy`, `quí`→`quý`, …)
- [x] Precomputed `first`/`last` syllable columns and out-degree table for O(1) engine lookups
- [x] Build fails loudly if entry count falls below a floor (40,000) or any row has fewer than 2 syllables
- [x] `--max-syllables` flag available (default: no cap) so a phrase cap can be applied later without a code change
**Non-functional**
- [x] Output DB is a few MB, opens read-only, no writes at runtime
- [x] License artifacts are correct and consistent across all five locations
## Architecture
**Source:** [`github.com/minhqnd/dictionary`](https://github.com/minhqnd/dictionary) release
**v2.0.0** → [`dictionary.db`](https://github.com/minhqnd/dictionary/releases/download/v2.0.0/dictionary.db),
**179 MB** (verified via the GitHub API; not in git; 357k+ entries, 1,500+ language pairs).
Code MIT, **data CC BY-SA 4.0**.
We do **not** republish a derived dictionary — every build downloads this asset and derives
locally. See `plan.md` → Data Distribution for the CI and Docker consequences.
**Derived schema** (`data/noitu.db`):
```sql
CREATE TABLE words (
word TEXT PRIMARY KEY, -- normalized, e.g. "pháp luật"
first TEXT NOT NULL, -- first syllable, "pháp"
last TEXT NOT NULL, -- last syllable, "luật"
syllables INTEGER NOT NULL -- 2, 3, 4, …
) WITHOUT ROWID;
CREATE INDEX idx_words_first ON words(first);
-- Out-degree per syllable: how many words start with it. Drives bot heuristics
-- and instant dead-end detection.
CREATE TABLE syllables (
syllable TEXT PRIMARY KEY,
out_degree INTEGER NOT NULL
) WITHOUT ROWID;
-- Accepted spelling variants that map to a canonical word. Built offline so the
-- runtime never has to reason about Vietnamese tone placement.
CREATE TABLE aliases (
variant TEXT PRIMARY KEY,
canonical TEXT NOT NULL REFERENCES words(word)
) WITHOUT ROWID;
CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT);
-- rows: source_url, source_license, built_at, word_count, builder_version
```
**Why an alias table instead of a runtime normalizer:** Vietnamese old-style vs new-style
tone placement (`hoà` vs `hòa`) produces genuinely different codepoint sequences. A runtime
algorithm to re-place tone marks is fiddly and easy to get subtly wrong; generating both
variants once at build time is data, cheap to fix, and trivially testable. KISS.
**Pipeline:**
```
dictionary.db ──► filter lang_code='vi'
──► NFC normalize (golang.org/x/text/unicode/norm) + lowercase + collapse spaces
──► keep len(strings.Fields(w)) >= 2 (and <= --max-syllables when set)
──► drop entries containing digits, latin-only tokens, or punctuation
──► dedupe
──► generate tone-placement variants → aliases
──► compute first/last/syllables + out_degree
──► write noitu.db + meta
──► assert: count >= 40000, every row >= 2 syllables, no orphan aliases
```
**Note on the graph:** allowing 3+ syllable words does not change the graph model — an edge
still runs `first ──word──► last`; it may simply span more syllables in between. Out-degree,
dead-end detection, and the engine's lookups are unaffected.
## Related Code Files
- Create: `server/cmd/build-dictionary/main.go` — CLI: `--in dictionary.db --out data/noitu.db [--max-syllables N]`
- Create: `server/cmd/build-dictionary/filter.go` — vi + ≥2-syllable + junk filters
- Create: `server/cmd/build-dictionary/aliases.go` — tone-placement variant generation
- Create: `server/cmd/build-dictionary/main_test.go` — filter/alias unit tests with fixtures
- Create: `data/LICENSE` — CC BY-SA 4.0 full text
- Create: `data/ATTRIBUTION.md`
- Create: `NOTICE`
- Create: `README.md` (replace stub) — quickstart, architecture, **License** section
- Create: `.gitignore` — `data/*.db` (covers both the 179 MB upstream and the derived DB), `web/node_modules`, `web/build`, server binaries
- Create: `Makefile` (or `Taskfile.yml`) — `make fetch-dict`, `make dict`, `make server`, `make web`, `make test`
- Create: `server/go.mod` — `module github.com/tiennm99dev/noitu/server`, Go 1.25+ (raised by modernc.org/sqlite; local toolchain is 1.26.5)
- Modify: none (repo is empty apart from `LICENSE` + `README.md`)
## Implementation Steps
1. `.gitignore`, `Makefile`, `server/go.mod` (`go mod init`), directory skeleton per plan layout.
2. Write `data/LICENSE` (verbatim CC BY-SA 4.0) and `data/ATTRIBUTION.md` naming: source repo + release URL, upstream sources (Wiktionary, `vntk/dictionary`), license URL, and the explicit modification list from `plan.md`.
3. Write `NOTICE` + README **License** section stating the split: Apache-2.0 code / CC BY-SA 4.0 data, and that the two are distributed as separate artifacts.
4. `server/cmd/build-dictionary`: open upstream DB read-only via `modernc.org/sqlite`, stream `vi` words.
5. Normalization: `norm.NFC`, `strings.ToLower`, `strings.Fields` → join with single space.
6. Filters: at least 2 fields (and ≤ `--max-syllables` when set); reject any token containing a digit, ASCII-only letters, or punctuation.
7. Alias generation: for each canonical word, emit old-style tone-placement variants for the `oa/oe/uy` nuclei; skip a variant if it collides with a different canonical word (log collisions).
8. Compute `first`, `last`, `syllables`, `out_degree`; write output DB in one transaction; write `meta` rows.
9. Assertions: word count ≥ 40,000; 100% of rows ≥ 2 syllables; every alias resolves; fail non-zero otherwise.
10. `make fetch-dict` downloads the 179 MB asset from the pinned release URL into `data/` (resumable, checksum-logged); `make dict` derives from it. Both documented in the README as one-time setup.
11. Unit tests on fixtures: normalization, ≥2-syllable filter (including a 3- and a 4-syllable entry accepted and a 1-syllable rejected), junk rejection, alias generation, collision handling.
## Success Criteria
- [x] `make fetch-dict && make dict` produces `data/noitu.db` from the upstream release
- [x] `sqlite3 data/noitu.db "SELECT COUNT(*) FROM words"` ≥ 40,000
- [x] Zero rows where `word` has fewer than 2 syllables; 3- and 4-syllable entries present
- [x] `hoà`, `thuý`, `quí` each resolve through `aliases` to a canonical word present in `words`
- [x] `meta` records source URL, license, build timestamp, word count
- [x] `data/LICENSE`, `data/ATTRIBUTION.md`, `NOTICE`, README license section all present and mutually consistent
- [x] `go test ./cmd/...` green
- [x] `data/*.db` is git-ignored
## Risk Assessment
| Risk | Signal | Response |
|---|---|---|
| Upstream schema differs from README-documented shape | Query errors on first run | Inspect actual tables with `.schema` first; adapt the extraction query — the pipeline stages after extraction are schema-independent |
| Fewer than 40k clean ≥2-syllable entries | Count assertion fails | Union with [Viet74K](https://vietnamese-wordlist.duyet.net/Viet74K.txt) filtered to ≥2 syllables; add it as a second `--in` source and extend `ATTRIBUTION.md` |
| 179 MB download is slow or the release URL moves | `make fetch-dict` fails or stalls | URL is pinned to release `v2.0.0`, download is resumable, and the derived DB is cached locally so the fetch is a one-time cost per contributor |
| Allowing 3+ syllables admits multi-word phrases that are not really words | Playtest complaints | `--max-syllables` flag already present; applying a cap is a rebuild, not a code change |
| Upstream data contains proper nouns / non-words that make the game feel wrong | Playtest complaints in later phases | Add a denylist file consumed by the builder; regenerate — no code change needed |
| Alias generation creates false accepts (a variant that is a different real word) | Collision log non-empty | Collisions are skipped by design and logged; review the log before release |
| CC BY-SA obligations misread | License review | Attribution + share-alike on the data artifact only; code untouched. `NOTICE` states the boundary explicitly |
plans/260904-1125-noi-tu-web-game/phase-02-go-dictionary-and-normalization.md
-150
View File
@@ -1,150 +0,0 @@
---
title: "Phase 2: Go Dictionary and Normalization"
status: done
phase: 2
priority: P1
effort: "2d"
dependencies: [1]
---
# Phase 2: Go Dictionary and Normalization
## Overview
The read-side of the dictionary: a Go package that normalizes arbitrary player input the
same way the builder normalized the corpus, and a read-only SQLite store exposing the three
lookups the game engine needs. Everything above this layer treats words as opaque strings.
## Requirements
**Functional**
- [x] `vietnamese.Normalize(raw) (word string, syllables []string, err error)` — NFC, lowercase, whitespace collapse, syllable split (any syllable count; length rules belong to the engine)
- [x] Store resolves a player-typed variant to its canonical word via `aliases`
- [x] `Store.Resolve(word)` → canonical word + exists
- [x] `Store.WordsStartingWith(syllable)` → words (for bot move generation)
- [x] `Store.OutDegree(syllable)` → int (0 means dead end)
- [x] Store opens the DB **read-only**; concurrent-safe for many goroutines
**Non-functional**
- [x] Lookup ≤ 1ms p99 under 100 concurrent readers
- [x] Normalization identical to the builder's — shared code path, not a reimplementation (DRY)
## Architecture
`internal/vietnamese` is imported by **both** `server/cmd/build-dictionary` and the server, so the
corpus and player input can never diverge. Phase 1 writes the builder against this package;
this phase hardens and tests it.
```go
// internal/vietnamese
func Normalize(raw string) (string, []string, error) // NFC + lower + collapse + split
const MinSyllables = 2
func HasEnoughSyllables(sylls []string) bool // len(sylls) >= MinSyllables
// internal/dictionary — loaded fully into memory at Open; no runtime SQL.
type Store struct{ /* maps, all written once during Open */ }
func Open(path string) (*Store, error) // file:...?mode=ro, closed before Open returns
func (s *Store) Resolve(word string) (canonical string, ok bool)
func (s *Store) LastSyllable(word string) (string, bool)
func (s *Store) WordsStartingWith(syl string) []string // shared slice, do not modify
func (s *Store) OutDegree(syl string) (int, error) // ErrNotFound if unknown
func (s *Store) RandomOpeningWord(minOutDegree int) (string, error)
func (s *Store) WordCount() int
func (s *Store) AliasCount() int
func (s *Store) License() string
```
**Revised during implementation — measured, not assumed.** The original design queried
SQLite per lookup and kept only `syllables` in memory. Measurement rejected that: a SQL
round-trip benchmarked at **55us** against **8.9ns** for a map hit. It would also have
threatened phase 3, where the hard bot explores hundreds of candidate moves inside a
150ms budget.
(An earlier draft justified this with a "p99 of 1.0046ms". That figure was wrong — the
Windows clock quantizes `time.Now()` to ~1ms, so it measured timer resolution, not the
code. Cite the benchmark, which does not time individual operations, and do not
reintroduce a latency test built on `time.Now()` around a nanosecond-scale call.)
The whole dictionary is now loaded at `Open` and the database closed immediately.
Measured on the real 48,216-word corpus: **~70ms load, ~7.8 MB heap**, and per operation,
all allocation-free:
| Operation | Cost | Allocations |
|---|---|---|
| `Resolve` | 8.9 ns | 0 |
| `WordsStartingWith` (full iteration) | 10.7 ns | 0 |
| `OutDegree` | 15 ns | 0 |
| `RandomOpeningWord` | 18 ns | 0 |
`Resolve` returns no error (a map lookup cannot fail) and `Close` is gone — there is
nothing left open. The result is both faster and simpler: no connection pool, no prepared
statements, no tail latency.
`WordsStartingWith` returns an `iter.Seq[string]`, not a slice. Returning the backing
slice let a caller sort, shuffle or `append` into dictionary state: verified on the real
corpus, a caller's write landed in the Store, 2,449 of 5,049 buckets had spare capacity
for `append` to scribble into, and `-race` confirmed the write/write race across
goroutines. An iterator removes the hazard structurally rather than by comment.
`Open` validates what it loaded against the builder's `meta.word_count`, cross-checks
every `syllables.out_degree` against the words actually indexed, and rejects orphan
aliases. Without that, a truncated database opens cleanly and the server starts, rejects
every word a player types, and fails every room creation.
**Resolve order:** exact hit in `words` → else `aliases` lookup → else not found.
**The engine chains on the canonical word, never on the alias the player typed.**
Canonicalization can move **either end** of the word. In the shipped dictionary roughly
half the aliases differ in the last syllable and more than a third in the first — typing
`sỹ hai` resolves to `sĩ hai`, moving the first syllable from `sỹ` to `sĩ`. An engine that
link-checked against the syllables the player typed would therefore **reject legal moves**.
`Resolve` returns the canonical form and `FirstSyllable`/`LastSyllable` report that form's
ends; phase 3 must use those, never `vietnamese.Normalize`'s split of the raw input.
All lookups are map reads; the database is closed before `Open` returns.
**Hot-path caching:** `OutDegree` is read on every bot move. Load the whole `syllables`
table into a `map[string]int` at `Open` (a few tens of thousands of entries, ~1MB) and serve
from memory. `WordsStartingWith` stays on SQL — the result sets are small and per-turn.
**Opening word selection:** pick uniformly from words whose `last` syllable has
`out_degree >= minOutDegree`, so a game never dies on move one.
## Related Code Files
- Create: `server/internal/vietnamese/normalize.go`
- Create: `server/internal/vietnamese/normalize_test.go`
- Create: `server/internal/dictionary/store.go`
- Create: `server/internal/dictionary/store_test.go`
- Create: `server/internal/dictionary/testdata/mini.db` — small fixture DB built by a test helper
- Modify: `server/go.mod` — add `modernc.org/sqlite`, `golang.org/x/text`
- Modify: `server/cmd/build-dictionary/main.go` — import `internal/vietnamese` instead of local normalization
## Implementation Steps
1. Implement `Normalize`: `norm.NFC.String` → `strings.ToLower` → `strings.Fields` → rejoin; return error on empty input.
2. Point the phase-1 builder at `internal/vietnamese` so one normalizer serves both sides.
3. `dictionary.Open`: DSN with `mode=ro`, verify `meta` table exists and log `word_count` + `source_license` at startup (license visibility).
4. Prepared statements for `Resolve` (words), `Resolve` (aliases), `WordsStartingWith`.
5. Load `syllables` into an in-memory map at open; `OutDegree` reads the map.
6. `RandomOpeningWord` via `ORDER BY RANDOM() LIMIT 1` over a joined out-degree filter.
7. Test helper that builds a tiny fixture DB in `t.TempDir()` from a hardcoded word set — no dependency on the real 60k DB in unit tests.
8. Tests: NFC equivalence (composed vs decomposed `ữ`), uppercase input, tabs/NBSP/multiple spaces, 1-syllable rejected while 3- and 4-syllable accepted, alias resolution for `hoà`/`thuý`/`quí`, unknown word, dead-end syllable returns 0.
9. Concurrency test: 100 goroutines × 1000 `Resolve` calls, `-race` clean.
## Success Criteria
- [x] `go test ./internal/... -race` green
- [x] Composed and decomposed spellings of the same word both resolve to one canonical entry
- [x] Words of 2, 3, and 4 syllables all resolve; a 1-syllable input is rejected by `HasEnoughSyllables`
- [x] `hoà lợi`-style tone variants resolve via `aliases`
- [x] Store refuses to open a missing or writable-mode DB path with a clear error
- [x] Startup log line names the data source and license
- [x] Builder and server share one normalization implementation (no duplicate NFC/lowercase logic)
## Risk Assessment
| Risk | Signal | Response |
|---|---|---|
| `modernc.org/sqlite` read performance disappoints under load | p99 lookup > 1ms in the concurrency test | Widen the in-memory cache: load `words` into a `map[string]string` too (~60k entries, a few MB) and use SQL only for cold paths |
| Normalization drift between builder and server | A word in the DB fails to match the same string typed by a player | Prevented structurally by the shared package; a regression test asserts round-trip on a sample of real DB rows |
| `strings.Fields` mishandles an exotic Unicode space | Rare rejection reports | `strings.Fields` already splits on all `unicode.IsSpace`; test explicitly covers NBSP and tab |
plans/260904-1125-noi-tu-web-game/phase-03-go-game-engine-and-bot-ai.md
-184
View File
@@ -1,184 +0,0 @@
---
title: "Phase 3: Go Game Engine and Bot AI"
status: done
phase: 3
priority: P1
effort: "4d"
dependencies: [2]
---
# Phase 3: Go Game Engine and Bot AI
## Overview
Pure, transport-free game logic: rule validation, per-game state, win/loss resolution, and
the bot's move selection at three difficulties. No WebSocket, no protobuf, no timers driven
by wall clock — the engine takes a deadline as data so it stays fully unit-testable.
## Requirements
**Functional**
- [x] `Engine.Submit(playerID, raw)` validates: at least 2 syllables → first syllable matches current → word exists → not already used
- [x] Distinct rejection reasons, not a boolean
- [x] Used-word set per game; a word is spent for both players
- [x] Detects `no legal move remains` for the player to act
- [x] Turn deadline held as `time.Time`, enforced by the caller; engine exposes `IsExpired(now)`
- [x] Bot difficulties: Easy (random), Medium (prefer low out-degree), Hard (dead-end win + depth-limited search)
- [x] Score model: points per accepted word, chain length tracked
**Non-functional**
- [x] Zero dependency on `wsapi` or generated protobuf types — engine speaks its own Go types
- [x] Hard bot move decision ≤ 150ms
- [x] Engine is safe under a single owning goroutine per game (documented; the room owns it)
## Architecture
The game is a **directed graph**: nodes = syllables, edges = words (`pháp ──"pháp luật"──► luật`).
A move traverses an unused edge from the current node. Formally this is *Directed Edge
Geography*, which is PSPACE-complete — so no perfect solver is attempted; the Hard bot uses
an instant-win check plus bounded search. Words of 3+ syllables are edges like any other —
they simply span extra syllables between `first` and `last`, so nothing in the graph model
changes.
```go
type RejectReason int
const (
ReasonNone RejectReason = iota
ReasonTooFewSyllables // fewer than vietnamese.MinSyllables (2)
ReasonWrongLink
ReasonNotInDictionary
ReasonAlreadyUsed
ReasonNotYourTurn
ReasonTimeout
)
type Engine struct {
dict *dictionary.Store
used map[string]struct{}
current string // syllable the next word must start with
turn PlayerID
deadline time.Time
history []Move
scores map[PlayerID]int
}
func New(dict *dictionary.Store, players []PlayerID, opening string, turnLimit time.Duration) (*Engine, error)
func (e *Engine) Submit(p PlayerID, raw string) (Move, RejectReason)
func (e *Engine) LegalMoves() ([]string, error) // for the player to act
func (e *Engine) HasLegalMove() (bool, error)
func (e *Engine) IsExpired(now time.Time) bool
func (e *Engine) Snapshot() State
```
**Validation order matters** — check cheapest and most-informative first so the player gets
the most useful message: turn → syllable count (≥2) → link → dictionary → reuse.
**Scoring and word length:** longer words are worth more, so allowing 3+ syllables adds a
reason to reach for them rather than being merely permissive. See the scoring formula below.
**Bot strategies** (`internal/bot`), all implementing one interface:
```go
type Strategy interface{ Choose(e *game.Engine) (string, error) }
```
| Difficulty | Behaviour |
|---|---|
| **Easy** | Uniform random legal move. Adds a 400-900ms simulated "thinking" pause so it feels human |
| **Medium** | Score each legal move by the out-degree of the syllable it hands the opponent; pick randomly among the lowest quartile. Never plays a guaranteed kill |
| **Hard** | 1) if any legal move lands on a syllable with 0 remaining continuations → play it (instant win). 2) else negamax with alpha-beta, depth 4, over the remaining edge set; eval = `-log(1 + opponent legal move count)`. 3) else fall back to Medium |
Search cost is bounded by move ordering (ascending opponent out-degree first) and a node
cap; the branching factor is the out-degree of visited syllables, typically < 50.
**Anti-frustration:** Hard plays the instant-win move only `hardKillRate` of the time
(a tunable constant, default 0.85). A bot that always wins is not a game.
**Scoring:** `10 + 2 × chainLength + 5 × (syllables − 2)` per accepted word, capped; final
score reported at game over. The syllable term rewards longer compounds now that they are
legal. Client persists a personal best in `localStorage` (phase 6) — no server storage.
## Revised during implementation — measured, not assumed
| Spec said | Now | Why |
|---|---|---|
| Validation order: turn → length → **link** → dictionary | turn → length → **dictionary** → link → reuse | Phase 2 established that 38% of aliases move the first syllable (`sỹ hai` → `sĩ hai`). The link can only be checked after resolving, or legal moves get rejected. Messages stay accurate either way |
| `dict *dictionary.Store` | `Dictionary` interface in `game` | Lets the engine and bot be tested on hand-built word graphs small enough to reason about, with no SQLite |
| `LegalMoves() ([]string, error)`, `HasLegalMove() (bool, error)` | no error returned | Phase 2's store is in-memory; these cannot fail |
| Medium **never plays a guaranteed kill** | Medium takes a win it can see | Simulation: withholding it made Medium lose to the **random** bot 97% of the time. A strategy that declines to win loses to a coin flip. Difficulty now comes from lookahead depth, not from refusing to play well |
| Eval `-log(1 + opponent move count)` | `+log(1 + moves)` at the node | Negamax evaluates from the perspective of the player *to move*, so the sign inverts. As specified, Hard searched for positions where it was about to be trapped and lost to Medium 71% of the time |
| Bot sleeps a "thinking" pause | `ThinkingDelay()` reported, caller schedules | Sleeping inside `Choose` would make every test wait in real time |
| Strategies take `*game.Engine` | `Board` interface (read-only) | A strategy can read the position but cannot play a move, so it cannot bypass `Submit` validation |
**Measured on the real 48,216-word corpus** (60 games per pairing, alternating sides):
| Matchup | Win rate | Mean game length |
|---|---|---|
| Hard vs Easy | 98% | 3.3 moves |
| Medium vs Easy | 87% | — |
| Hard vs Medium | **65%** | 2.9 moves |
| Easy vs Easy | — | 15.3 moves |
Hard decision latency on the real corpus: p95 12.5ms, **max 18.6ms** against a 150ms budget.
Sides alternate because moving first is an advantage in its own right, and fixed seating
reports that advantage as skill — on a 7-syllable graph with fixed sides the Hard-vs-Medium
figure was exactly 50%, pure first-move effect.
**How much lookahead is worth is a property of the graph, not of the bots.** The synthetic
ladder gave 78% on one seed and 49% on another, so the synthetic test now only guards
against a gross regression (>= 45%) and the real-corpus test carries the claim. Earlier
drafts of this document quoted the synthetic 78% and a 6ms max; both were specific to a
single run and are corrected above.
**Playability signal for phase 6 — the dominant characteristic, not an edge case.** Games
against Hard end in ~3 moves against ~15 for two random bots. The dictionary has 1,814
dead-end syllables, so an instant kill is available at roughly 42% of opening positions and
Hard simply takes one; the search runs in only about half of its decisions. `hardKillRate`
(0.85) softens this but does not remove it. Needs playtesting before release; the levers are
a lower kill rate and dead-end-aware opening selection, both tunable constants.
## Related Code Files
- Create: `server/internal/game/engine.go`
- Create: `server/internal/game/state.go` — `Move`, `State`, `PlayerID`, `RejectReason`
- Create: `server/internal/game/engine_test.go`
- Create: `server/internal/bot/bot.go` — `Strategy` interface, difficulty registry, thinking delay
- Create: `server/internal/bot/strategy_easy.go`
- Create: `server/internal/bot/strategy_medium.go`
- Create: `server/internal/bot/strategy_hard.go`
- Create: `server/internal/bot/bot_test.go`
- Create: `server/internal/bot/simulate_test.go` — 100-game difficulty ladder assertion
## Implementation Steps
1. `state.go`: `PlayerID`, `Move{Player, Word, First, Last, Syllables, Points, At}`, `RejectReason` with `String()`, `State` snapshot struct.
2. `engine.New`: seed `used` with the opening word, set `current` to its last syllable, assign first turn.
3. `Submit`: normalize via `vietnamese.Normalize`, then run the validation order above; on success record the move, add to `used`, advance `current` and `turn`, reset deadline, add points.
4. `LegalMoves`: `dict.WordsStartingWith(current)` minus `used`. `HasLegalMove` short-circuits on the first hit rather than materializing the slice.
5. Engine unit tests, one per rejection reason, plus: reuse blocked across both players, chain advances correctly, no-legal-move detection, `IsExpired` boundary.
6. `bot.Strategy` interface + registry keyed by difficulty; shared randomized thinking delay helper.
7. Easy: uniform pick. Medium: out-degree-of-result scoring, lowest-quartile random pick, explicit guard that skips a 0-out-degree kill.
8. Hard: instant-win scan gated by `hardKillRate`; negamax + alpha-beta depth 4 with move ordering and node cap; Medium fallback.
9. `simulate_test.go`: 100 headless bot-vs-bot games per pairing; assert Hard beats Easy well above chance and Medium sits between. Deterministic via a seeded RNG.
10. Benchmark Hard's `Choose` to confirm ≤150ms on the real DB.
## Success Criteria
- [x] `go test ./internal/game/... ./internal/bot/... -race` green
- [x] Every `RejectReason` has a test that produces exactly it
- [x] A 1-syllable submission yields `ReasonTooFewSyllables`; 3- and 4-syllable words are accepted and score the syllable bonus
- [x] A word played by either player cannot be replayed by the other
- [x] `HasLegalMove` returns false on a synthetic dead-end board and the engine reports the correct winner
- [x] Hard win rate vs Easy > 70% over 100 seeded games; Medium strictly between Easy and Hard
- [x] `BenchmarkHardChoose` ≤ 150ms/op against the real `noitu.db`
- [x] Engine package imports no transport or protobuf package (verified by an import assertion test)
## Risk Assessment
| Risk | Signal | Response |
|---|---|---|
| Depth-4 search too slow on hub syllables (high out-degree) | Benchmark exceeds 150ms | Node cap already present; reduce to depth 3 for syllables with out-degree > 200 — the depth is a tunable constant |
| Hard bot feels unbeatable | Playtest win rate ≈ 0% | Lower `hardKillRate`; it exists precisely for this dial |
| Bot picks obscure words that feel unfair | Playtest complaints | Out of scope for v1 (no frequency data in the derived DB); note as a post-v1 item requiring a frequency column |
| Engine mutated from two goroutines | `-race` failures once rooms land in phase 5 | Ownership rule documented here and enforced in phase 5: exactly one goroutine per room owns its engine |
plans/260904-1125-noi-tu-web-game/phase-04-protobuf-contract-and-codegen.md
-226
View File
@@ -1,226 +0,0 @@
---
title: "Phase 4: Protobuf Contract and Codegen"
status: done
phase: 4
priority: P1
effort: "2d"
dependencies: [1]
---
# Phase 4: Protobuf Contract and Codegen
## Overview
One `.proto` file defines every message crossing the WebSocket, and `buf` generates both the
Go server types and the JavaScript client types from it. No hand-written message structs on
either side — the schema is the single source of truth for the wire contract.
Independent of phases 2-3; can be built in parallel with them.
## Requirements
**Functional**
- [x] `proto/noitu/v1/game.proto` covers handshake with nickname, create/join room, start bot game, submit word, move results, turn state, game over, errors, heartbeat
- [x] `buf generate` emits Go into `server/gen/noitu/v1` and JS into `web/src/lib/proto`
- [x] Protocol version field in the handshake — `Hello.protocol_version`, `Welcome.protocol_version`, and `wsapi.ProtocolVersion = 1`. Rejecting a mismatch lands in phase 5, which is where `Hello` is first read
- [x] Generated code is committed (reviewable diffs, no toolchain needed to build)
**Non-functional**
- [x] JS bundle cost of the protobuf runtime kept small — ESM, tree-shakeable. Measured against a budget in phase 6, where there is a bundle to measure
- [x] `buf lint` and `buf breaking` (against `main`) run in CI
## Architecture
**Tooling:** [`buf`](https://buf.build) drives both targets from one config.
- Go: `protoc-gen-go` (messages only — no gRPC, this is raw WS framing)
- JS: [`@bufbuild/protobuf`](https://github.com/bufbuild/protobuf-es) **v2.14.1** +
`@bufbuild/protoc-gen-es` **v2.14.1** (both verified current on npm), generated with
`target=js` plus `.d.ts`. Chosen over `protobuf.js`: it is the only fully
conformance-compliant JS implementation, ships ESM for tree-shaking, and produces a much
smaller browser bundle.
> `buf` **is** installed (v1.69.0), retiring the plan-time note that it was missing. It stays
> optional for contributors either way: generated code is committed, so `buf` is required only
> when the schema changes.
**Framing:** every WS frame is a binary message — `ClientMessage` client→server,
`ServerMessage` server→client, each a single `oneof`. No JSON fallback, no text frames.
```proto
syntax = "proto3";
package noitu.v1;
option go_package = "github.com/tiennm99dev/noitu/server/gen/noitu/v1;noituv1";
enum Difficulty { DIFFICULTY_UNSPECIFIED = 0; EASY = 1; MEDIUM = 2; HARD = 3; }
enum RejectReason {
REJECT_UNSPECIFIED = 0;
TOO_FEW_SYLLABLES = 1; // fewer than 2 syllables
WRONG_LINK = 2;
NOT_IN_DICTIONARY = 3;
ALREADY_USED = 4;
NOT_YOUR_TURN = 5;
TIMEOUT = 6;
}
enum GameEndReason {
END_UNSPECIFIED = 0;
END_TIMEOUT = 1;
END_NO_LEGAL_MOVE = 2;
END_OPPONENT_LEFT = 3;
END_RESIGNED = 4;
}
message Hello { uint32 protocol_version = 1; string resume_token = 2; string nickname = 3; }
message StartBotGame { Difficulty difficulty = 1; }
message CreateRoom {}
message JoinRoom { string room_code = 1; }
message SubmitWord { string word = 1; uint32 turn_seq = 2; } // turn_seq guards double-submits
message Resign {}
message Ping { int64 client_time_ms = 1; }
message ClientMessage {
oneof payload {
Hello hello = 1; StartBotGame start_bot_game = 2; CreateRoom create_room = 3;
JoinRoom join_room = 4; SubmitWord submit_word = 5; Resign resign = 6; Ping ping = 7;
}
}
// accepted_nickname is what the server actually stored after sanitization — the client
// must display this, not the string it sent.
message Welcome { string session_id = 1; string resume_token = 2; uint32 protocol_version = 3; string accepted_nickname = 4; }
message RoomCreated { string room_code = 1; }
message RoomJoined { string room_code = 1; string opponent_name = 2; }
message PlayedWord { string word = 1; bool by_me = 2; uint32 points = 3; uint32 syllables = 4; }
message GameStarted {
string opening_word = 1; string current_syllable = 2; bool my_turn = 3;
int64 deadline_unix_ms = 4; uint32 turn_seq = 5; uint32 turn_limit_ms = 6;
}
message TurnUpdate {
PlayedWord played = 1; string current_syllable = 2; bool my_turn = 3;
int64 deadline_unix_ms = 4; uint32 turn_seq = 5;
uint32 my_score = 6; uint32 opponent_score = 7; uint32 chain_length = 8;
}
message MoveRejected { RejectReason reason = 1; string word = 2; uint32 turn_seq = 3; }
message GameOver { bool i_won = 1; GameEndReason reason = 2; uint32 my_score = 3; uint32 chain_length = 4; }
message OpponentLeft { bool can_reconnect = 1; uint32 grace_ms = 2; }
message ServerError { string code = 1; string message = 2; } // message is a UI key, not prose
message Pong { int64 client_time_ms = 1; int64 server_time_ms = 2; }
message ServerMessage {
oneof payload {
Welcome welcome = 1; RoomCreated room_created = 2; RoomJoined room_joined = 3;
GameStarted game_started = 4; TurnUpdate turn_update = 5; MoveRejected move_rejected = 6;
GameOver game_over = 7; OpponentLeft opponent_left = 8; ServerError error = 9; Pong pong = 10;
}
}
```
**Design notes**
- `TurnUpdate` is sent to **both** players after every accepted move, with `by_me`/`my_turn`
rendered per recipient — the server serializes one message per player, not a broadcast.
- `deadline_unix_ms` is an absolute server timestamp; the client renders a countdown from it.
`Ping`/`Pong` carry both clocks so the client can correct for offset. Never trust the client clock.
- `turn_seq` on `SubmitWord` makes double-submits and late submissions detectable server-side.
- `ServerError.message` carries a **UI key** (e.g. `room_not_found`), not user-facing prose —
the Vietnamese strings live in the frontend so all copy stays in one place.
- `Hello.nickname` is a **request**, not a fact. The server sanitizes it (phase 5) and returns
the stored value in `Welcome.accepted_nickname`; `RoomJoined.opponent_name` is always a
server-sanitized string. A client must never render another player's raw input.
- The engine's `RejectReason` and this proto enum are deliberately separate types with an
explicit mapping function; the wire contract must not be hostage to internal refactors.
## Related Code Files
- Create: `proto/noitu/v1/game.proto`
- Create: `buf.yaml`, `buf.gen.yaml`, `buf.lock`
- Create: `server/gen/noitu/v1/game.pb.go` (generated, committed)
- Create: `web/src/lib/proto/game_pb.js` (generated, committed)
- Create: `server/internal/wsapi/convert.go` — engine ↔ proto enum mapping + tests
- Modify: `Makefile` — `make proto`
- Modify: `server/go.mod` — `google.golang.org/protobuf`
- Modify: `web/package.json` — `@bufbuild/protobuf`, dev dep `@bufbuild/protoc-gen-es`
## Implementation Steps
1. `buf.yaml` (module + lint/breaking config) and `buf.gen.yaml` with the two plugins and their output dirs.
2. Write `game.proto` as above; `buf lint` clean.
3. `make proto` → `buf generate`; commit both generated trees.
4. `convert.go`: `game.RejectReason` → `noituv1.RejectReason`, engine end reasons → `GameEndReason`. Exhaustive `switch` with a default that returns the UNSPECIFIED value **and** logs — a silently-dropped new reason is a bug.
5. Round-trip test in Go: marshal each `ServerMessage` variant, unmarshal, assert equality.
6. Round-trip test in JS (Vitest): decode a fixture emitted by the Go test, assert field values — proves cross-language wire compatibility, not just self-consistency.
7. CI step: `buf lint`, `buf breaking --against '.git#branch=main'`, and a check that regenerating produces no diff (generated code stays in sync).
## Success Criteria
- [x] `make proto` regenerates both targets with zero diff on a clean tree
- [x] `buf lint` clean; `buf breaking` wired into CI
- [x] Go round-trip test covers every `oneof` variant in both directions
- [x] JS decodes a Go-produced binary fixture and reads correct values
- [x] `convert.go` mapping is exhaustive; a test fails if an engine reason gains a value with no proto counterpart
- [x] No hand-written message struct exists in `server/` or `web/`
## Risk Assessment
| Risk | Signal | Response |
|---|---|---|
| Schema churn breaks an already-deployed client | Decode errors after a deploy | Additive-only changes, never reuse a tag, `reserved` on removals; `protocol_version` in `Hello` lets the server reject incompatible clients with a clear message instead of failing obscurely |
| `buf` unavailable in a contributor's environment | `make proto` fails locally | Generated code is committed, so building and running never requires `buf` — only changing the schema does |
| Protobuf runtime inflates the JS bundle | Bundle budget exceeded in phase 6 | `protobuf-es` is ESM/tree-shakeable and only the generated messages are imported; measure in phase 6 and drop to hand-rolled binary framing only if it genuinely fails the budget |
| Enum drift between engine and wire | A new reject reason silently arrives as UNSPECIFIED | Exhaustive-switch test plus a logged default in `convert.go` |
## Phase 4 Outcome (2026-09-04)
Delivered and verified end to end. `buf lint` clean under the `STANDARD` category with no
carve-outs, regeneration byte-identical, `go test ./... -race` green across every package,
and 23 JavaScript assertions decoding bytes the Go suite produced.
**Two deviations from the sketch above, both accepted before implementation**
| Sketch | Shipped | Why |
|---|---|---|
| `TOO_FEW_SYLLABLES`, `EASY`, `END_TIMEOUT` | `REJECT_REASON_TOO_FEW_SYLLABLES`, `DIFFICULTY_EASY`, `GAME_END_REASON_TIMEOUT` | buf's `STANDARD` lint mandates the enum-name prefix. Keeping the short names would have meant an `ENUM_VALUE_PREFIX` exception, weakening "buf lint clean" to "clean with a carve-out". `protobuf-es` strips the prefix, so the JS side still reads `RejectReason.WRONG_LINK` |
| `web/` untouched until phase 6 | `web/` bootstrapped as a bare npm package | The phase's own JS codegen and Vitest cross-language test have nowhere to live otherwise. Protobuf + Vitest only — no SvelteKit, no routes, no config |
**Additions beyond the sketch**
- `PlayedWord.typed` (field 5). `game.Move` carries both the canonical spelling and the raw
input, and the UI has to show that a correction happened rather than appearing to rewrite
the player's text.
- `REJECT_REASON_GAME_OVER` (7). `game.ReasonGameOver` exists in the engine; without a wire
counterpart the exhaustiveness test fails, which is the test working.
- `wsapi.ProtocolVersion = 1`, giving `protocol_version` a defined value rather than a field
nobody sets.
**Toolchain**
Both code generators resolve from the repo's own manifests — a `tool` directive in
`server/go.mod` and `web/package.json` — so regenerating needs only `buf`, with no globally
installed plugin binaries. `buf.gen.yaml` sets `clean: true`, so the JavaScript wire test
lives in `web/tests/`, outside the generated tree that gets wiped on each run.
**Cross-language checking**
`server/internal/wsapi/wire_test.go` writes 17 binary fixtures to `proto/testdata/` under
`-update`; both suites then read those same bytes. A JS-only round trip would only have
proven that runtime self-consistent, which is exactly the failure that ships a broken client.
`.gitattributes` marks the fixtures `binary`: nine of them contain `0x0a` and no NUL, so a
checkout with `autocrlf` enabled could otherwise rewrite them into messages that no longer
decode.
**Review found four guards that reported green while protecting nothing.** All fixed and
then negative-tested by deliberately breaking each one:
| Defect | Fix |
|---|---|
| `buf breaking --against '.git#branch=main'` fails on every PR — a `pull_request` checkout is a detached HEAD with no local `main` | `.git#ref=origin/main` |
| `git diff --exit-code` never reports untracked files, so a plugin emitting a *new* file would pass the sync check | `git add --intent-to-add` first, in both CI and `make proto-check` |
| The oneof-coverage test compared the schema against a hand-written list of arm names, which drifts in lockstep with the table it duplicates | Coverage now derived from what each sample message actually sets, via `WhichOneof` |
| The exhaustiveness walk ended at `String() == "unknown"`, so a reason added without a `String` case shrank the range under test and still passed | `game.NumRejectReasons` / `game.NumEndReasons` sentinels, plus a test that every reason is named |
`WireDifficulty` was also removed: it had no caller and no message in the schema carries a
`Difficulty`, so its doc comment described a capability the contract cannot express.
**Deferred to phase 5**, where `Hello` is first read: rejecting a mismatched
`protocol_version`, and nickname sanitization behind `Welcome.accepted_nickname`.
plans/260904-1125-noi-tu-web-game/phase-05-go-websocket-server-and-rooms.md
-204
View File
@@ -1,204 +0,0 @@
---
title: "Phase 5: Go WebSocket Server and Rooms"
status: done
phase: 5
priority: P1
effort: "4d"
dependencies: [3, 4]
---
# Phase 5: Go WebSocket Server and Rooms
## Overview
The transport and orchestration layer: a WebSocket endpoint that speaks Protobuf frames,
a hub owning all live rooms, per-room goroutines driving the engine and turn timers, and
the bot wired in as a virtual player. After this phase the whole game is playable over
`websocat` with no frontend.
## Requirements
**Functional**
- [x] `GET /ws` upgrades and runs the session loop; `GET /healthz` for liveness
- [x] Bot rooms (1 human + bot) and PvP rooms (2 humans, joined by 6-char room code)
- [x] Nicknames sanitized server-side and echoed back; opponents only ever see sanitized values
- [x] Server-authoritative 20s turn timer (`NOITU_TURN_LIMIT`); expiry ends the game for the player on turn
- [x] Every accepted move fans out a per-recipient `TurnUpdate`
- [x] Bot moves scheduled on the room goroutine after a randomized thinking delay
- [x] Session resume token issued in `Welcome`, honoured on reconnect within a grace window
- [x] Serves the built frontend as static files so one binary is the whole deployment
**Non-functional**
- [x] One goroutine owns each room's engine — no shared mutable game state (`-race` clean)
- [x] Per-connection read limit, read/write deadlines, and rate limiting on `SubmitWord`
- [x] Graceful shutdown drains rooms and tells clients why
## Architecture
```
┌──────── hub (mutex-guarded maps) ────────┐
conn ──► session ──chan──► room goroutine ──► game.Engine
│ │ └► bot.Strategy
│ └── time.Timer (turn deadline)
└── writer goroutine (owns the socket write side)
```
**Concurrency contract — the core invariant of this phase:**
- Each connection has exactly **one reader goroutine** and **one writer goroutine**. Nothing
else touches the socket. (`coder/websocket` tolerates concurrent writes, but a single
writer keeps ordering deterministic.)
- Each room has exactly **one goroutine** that owns its `game.Engine`. All input arrives as
messages on the room's channel. The engine is never locked because it is never shared.
- The hub owns only the `code → *room` and `sessionID → *session` maps, guarded by a mutex.
It never touches engine state.
**Room lifecycle**
```
CreateRoom → generate code (6 chars, unambiguous alphabet: no 0/O/1/I) → room waits
JoinRoom → second player attaches → engine created → GameStarted to both
StartBotGame→ room created immediately with a bot as player 2 → GameStarted
in play → SubmitWord | timer fire | disconnect
GameOver → both notified → room lingers briefly for a rematch, then hub evicts it
```
**Turn timer:** the room goroutine holds a `time.Timer` for the current deadline, selected
on alongside the input channel. On fire: mark the player on turn as the loser
(`END_TIMEOUT`) and broadcast `GameOver`. The absolute deadline also goes to clients in
`TurnUpdate.deadline_unix_ms` — clients render a countdown but never decide the outcome.
**Bot integration:** the bot is a player ID with no connection. When it is the bot's turn,
the room schedules `Choose` on a worker and delivers the result back through the same input
channel as a human move, so it flows through identical validation. If the bot has no legal
move, the human wins.
**Reconnect:** `Welcome` carries a `resume_token`. On disconnect, the room keeps the seat
open for `graceMs` (default 30s) and sends the opponent `OpponentLeft{can_reconnect}`. A
`Hello` carrying a valid token rebinds the seat and replays current state via `GameStarted`
+ the latest `TurnUpdate`. Timeout ends the game as `END_OPPONENT_LEFT`.
**Security / abuse limits**
- `conn.SetReadLimit(4096)` — no legitimate message approaches this (verified API: `SetReadLimit(n int64)`)
- Read deadlines come from `context.WithTimeout` passed to `conn.Read` — `coder/websocket`
has **no** `SetReadDeadline`; everything is context-driven. Ping via `conn.Ping(ctx)` every
20s, treating a ctx timeout as a miss and closing after 2
- `SubmitWord` rate-limited to ~5/sec per session (a submit is one dictionary lookup)
- Room codes from `crypto/rand`; join attempts rate-limited to make brute-forcing pointless
- Origin check via `websocket.AcceptOptions.OriginPatterns` from `NOITU_ALLOWED_ORIGINS`;
never set `InsecureSkipVerify` outside local dev
- **Nickname sanitization** (`sanitizeNickname`): NFC normalize, strip control characters and
zero-width joiners, collapse whitespace, trim, cap at 20 runes, reject empty → assign
`Người chơi N`. Nicknames are shown to strangers, so this is a real input-validation
boundary, not cosmetic. A denylist hook is left in place for post-v1 if abuse appears
- Every rejection returns a UI key, never a raw internal error string
## Related Code Files
- Create: `server/cmd/noitu-server/main.go` — flags/env, dict open, hub, HTTP mux, graceful shutdown
- Create: `server/internal/wsapi/server.go` — upgrade handler, origin check, static file serving
- Create: `server/internal/wsapi/session.go` — reader/writer goroutines, codec, resume tokens
- Create: `server/internal/wsapi/hub.go` — room + session registries, room code generation
- Create: `server/internal/wsapi/room.go` — room goroutine, timer, bot scheduling, fan-out
- Create: `server/internal/wsapi/codec.go` — protobuf marshal/unmarshal over binary frames
- Create: `server/internal/wsapi/ratelimit.go`
- Create: `server/internal/wsapi/nickname.go` — `sanitizeNickname` + tests
- Create: `server/internal/wsapi/room_test.go`, `session_test.go`, `hub_test.go`
- Modify: `server/go.mod` — `github.com/coder/websocket`
- Modify: `Makefile` — `make server`, `make run`
## Implementation Steps
1. `main.go`: config from env (`NOITU_ADDR`, `NOITU_DB_PATH`, `NOITU_TURN_LIMIT` default `20s`, `NOITU_ALLOWED_ORIGINS`, `NOITU_WEB_DIR`), open the dictionary read-only, construct hub, mount `/ws`, `/healthz`, and static assets, `signal.NotifyContext` shutdown. **Log the dictionary's `WordCount()`, `AliasCount()` and `License()` at startup** — the store deliberately does not log (a library writing to the global logger fights phase 5's structured logging), so the CC BY-SA attribution surfaces here or nowhere.
2. `codec.go`: `Encode(*ServerMessage) []byte` / `Decode([]byte) (*ClientMessage, error)`; binary message type only, reject text frames.
3. `session.go`: reader goroutine (read limit, per-read `context.WithTimeout`, decode, forward to the room or hub) and writer goroutine (buffered channel, single owner of writes). Generate `session_id` and `resume_token` with `crypto/rand`; sanitize `Hello.nickname`; send `Welcome` carrying `accepted_nickname`; reject mismatched `protocol_version`.
4. `hub.go`: registries + mutex; `CreateBotRoom`, `CreateRoom`, `JoinRoom(code)`, `ResumeSession(token)`, eviction of finished/idle rooms via a janitor ticker.
5. `room.go`: the goroutine — `select` over input channel, turn timer, and context done. Handle `SubmitWord` (validate turn_seq, call engine, fan out per-recipient `TurnUpdate` or `MoveRejected`), `Resign`, disconnect, timer fire, bot scheduling, and no-legal-move detection after every move.
6. Per-recipient rendering: build `TurnUpdate` twice, flipping `by_me`/`my_turn` and swapping scores. Do not broadcast one shared message.
7. Bot wiring: on bot turn, run `Choose` on a worker goroutine with the room's context, deliver via the input channel after the thinking delay; cancel cleanly if the room ends first.
8. Reconnect: grace timer per seat, `OpponentLeft` to the other player, state replay on successful resume, `END_OPPONENT_LEFT` on grace expiry.
9. `ratelimit.go`: token bucket per session for `SubmitWord` and per IP for room joins.
10. Tests with in-process WS clients: full bot game to completion; two clients play a PvP game; timeout ends the game; reuse and wrong-link rejections reach the client with the right enum; disconnect+resume within grace restores state; resume after grace fails cleanly. All under `-race`.
11. Manual smoke via `websocat` documented in the README.
## Success Criteria
- [x] `go test ./internal/wsapi/... -race` green
- [x] A full bot game is playable end to end over a raw WS client, no frontend involved
- [x] Two in-process clients complete a PvP game with correct alternating turns and a correct winner
- [x] Turn timeout ends the game server-side even if the client sends nothing
- [x] Disconnect + resume within 30s restores the game; after 30s the opponent wins by `END_OPPONENT_LEFT`
- [x] Read limit, rate limit, and origin check each covered by a test
- [x] `sanitizeNickname` tested against: over-length, control chars, zero-width chars, whitespace-only, and empty input; opponents never receive an unsanitized string
- [x] `go build ./cmd/noitu-server` yields one binary that serves both `/ws` and the static frontend
- [x] Graceful shutdown notifies connected clients rather than dropping sockets silently
## Risk Assessment
| Risk | Signal | Response |
|---|---|---|
| Engine touched from two goroutines | `-race` failure | The one-goroutine-per-room rule is structural: the engine is only reachable through the room's channel. A review checklist item, not a runtime guard |
| Timer and message race (move accepted exactly at the deadline) | Flaky test, disputed loss | Resolve entirely inside the room goroutine: the timer fire and the move arrive on the same `select`, so one strictly precedes the other. `turn_seq` makes the outcome explicit to the client |
| Bot goroutine outlives its room | Goroutine leak under load test | Room context cancels the worker; a leak test asserts goroutine count returns to baseline after N games |
| Room code collision | Join lands in the wrong room | Generate with `crypto/rand` and retry on collision inside the hub mutex |
| Restart drops all live games | Players lose in-progress matches on deploy | Accepted for v1 (documented in `plan.md`); graceful shutdown sends a typed error so the UI can explain it |
| Offensive nicknames shown to strangers | Abuse reports | Sanitization caps length and strips control/zero-width characters; a denylist hook is already in `nickname.go` for a data-only fix if it becomes a real problem |
| Reconnect replay diverges from actual state | Resumed client shows a stale board | Replay is a fresh snapshot from the engine, never a stored copy of past messages |
## Phase 5 Outcome (2026-09-05)
Delivered. `go vet` and `go test ./... -race` green across every package; the whole game
is playable over a raw WebSocket with no frontend, and the binary was run against a
fixture dictionary to confirm startup, `/healthz`, the handshake, ping/pong, room
creation and the error keys end to end.
**Two deviations from the spec above, both accepted before implementation**
| Spec | Shipped | Why |
|---|---|---|
| Bot worker searches the live engine | Worker gets an immutable `frozenBoard` — legal moves, a copy of the used set, and the read-only dictionary | `bot.Board.Used` reads engine state, so handing a worker the live engine races every resign and disconnect the room processes while the bot thinks |
| Per-read `context.WithTimeout` on `conn.Read` | No read deadline; liveness is `conn.Ping` every 20s, 10s timeout, close after 2 misses | A read deadline cannot tell a healthy player idling in the lobby from a dead socket, and would disconnect the first |
`NewServer` takes a `Dictionary` interface (the engine's contract plus
`RandomOpeningWord`) rather than `*dictionary.Store`, so tests play whole games against a
four-word graph whose outcome a reader can verify by eye. Tests live in `wsapi_test.go`
and `regression_test.go` rather than the three files sketched above.
**Bugs found by writing the tests**, before review:
| Defect | Consequence |
|---|---|
| Resume token unregistered the instant the socket dropped | A reconnect could never find its seat — the feature was entirely dead |
| `writeLoop` took its write deadline from the cancelled session context | Every refusal queued just before a close was silently discarded |
| `keyedLimiter.sweep` required a full bucket, but tokens refill lazily | Nothing was ever swept: unbounded map growth keyed by whoever connects |
**Review found three more, all proven with repros and all fixed:**
| Defect | Fix |
|---|---|
| The hub bound a joiner to seat `p2` *before* the room decided whether to seat them, and neither `Submit` nor `Resign` checked that the sender owned the seat. Anyone holding a room code — which is pasted into group chats by design — could resign or play on a seated player's behalf | Seating moved onto the room goroutine and happens only on success; `occupies` verifies the sending connection is the one in the seat |
| Cancelling the session context hard-closes the socket under `coder/websocket`, so the shutdown notice never left. Acceptance criterion (i) was failing | Read context separated from the teardown signal and rooted at `Background`, with a watcher that flushes before dropping the socket |
| A room whose game never started had no turn timer and no opponent, so nothing could ever end it; room creation was unlimited | A pre-game disconnect cancels the room; creation is rate-limited per session |
Fixing those exposed a fourth, in `room.send`: a buffered channel and a cancelled context
are both ready, so `select` chose at random and half of all sends into a finished room
reported success. The caller then waited forever for a reply from a goroutine that had
stopped reading. The done-check is its own statement now.
Also corrected: the join limiter keyed on the session id, which a client resets by
reconnecting — it keys on the client IP now, deliberately ignoring `X-Forwarded-For`,
which is attacker-controlled until phase 7 pins a trusted proxy. The static-file guard
compared string prefixes, so `/srv/webhooks` counted as inside `/srv/web`; it is a real
path-boundary check now. Room codes use rejection sampling instead of a modulo, which was
making the first eight letters 12% more likely. The handshake is one-shot.
`sanitizeNickname`'s seat parameter was dead — both players who sent an empty name became
"Người chơi 1" — so the collision is resolved in the room, where both names are known,
and now also covers two players who *choose* the same name.
Every fix has a regression test, and the important ones were mutation-tested by
reintroducing the defect: the authorization guard fails 10/10 against the old code, the
shutdown guard 7/10 and the `room.send` guard 6/10 — both of those are timing races, so
detection is probabilistic by nature.
**Deferred to phase 7:** the trusted-proxy policy for `X-Forwarded-For`, and a room
identity separate from the code if rematch is added, since codes are recycled on eviction.
plans/260904-1125-noi-tu-web-game/phase-06-sveltekit-frontend.md
-275
View File
@@ -1,275 +0,0 @@
---
title: "Phase 6: SvelteKit Frontend"
status: done
phase: 6
priority: P1
effort: "5d"
dependencies: [4, 5]
---
# Phase 6: SvelteKit Frontend
## Overview
The playable UI: a SvelteKit SPA in JavaScript that connects over WebSocket, speaks
Protobuf, and renders the game. Vietnamese throughout, dark mode, local high score. This
phase delivers the complete vs-bot experience; phase 7 layers the online 1v1 flows on top.
## Requirements
**Functional**
- [x] Home screen: nickname input, play vs bot (Easy / Trung bình / Khó), or go to online play
- [x] Nickname persisted in `localStorage`, sent in `Hello`, and replaced by `accepted_nickname` from `Welcome`
- [x] Game screen: chain history, current syllable prompt, input, countdown ring, both scores
- [x] Rejections shown as specific Vietnamese messages, mapped from `RejectReason`
- [x] Game-over screen: result, final score, new-personal-best marker, rematch / home
- [x] Dark mode toggle, persisted; respects `prefers-color-scheme` on first visit
- [x] Personal best per difficulty in `localStorage`
- [x] Connection status indicator; automatic reconnect with backoff
- [x] In-app footer credit for the CC BY-SA 4.0 dictionary source, with links
**Non-functional**
- [x] `adapter-static` build, served by the Go binary
- [ ] Mobile-first; usable one-handed on a phone with the keyboard up
- [x] No wordlist or dictionary data in the client bundle
- [ ] Vietnamese diacritic input (Telex/VNI IMEs) works — no input interception that breaks composition
## Architecture
**Stack:** SvelteKit 2 + Svelte 5 (runes: `$state`, `$derived`, `$effect`), plain JavaScript
with JSDoc types, `adapter-static`, Vite. Scaffold with `npx sv create`.
```
web/src/
lib/
proto/ # generated (phase 4) — do not hand-edit
ws/client.js # socket lifecycle, protobuf codec, reconnect backoff, ping/pong
ws/messages.js # thin senders: startBotGame(), submitWord(), createRoom(), joinRoom()
stores/game.svelte.js # $state game model, fed only by ServerMessage
stores/settings.svelte.js# nickname + theme + high scores, localStorage-backed
i18n/vi.js # ALL user-facing strings, incl. RejectReason → message map
components/ # ChainHistory, WordInput, CountdownRing, ScoreBoard, DifficultyPicker,
# ConnectionBadge, GameOverPanel, ThemeToggle, AttributionFooter
routes/
+layout.svelte # theme application, footer
+page.svelte # home
play/+page.svelte # game screen (bot and PvP share it)
online/+page.svelte # room create/join (phase 7 fills this in)
```
**State rule:** the store is a projection of server messages. The client never decides
whether a word is valid, whose turn it is, or who won — it renders what the server sent.
The only client-owned state is theme, personal best, and the input box.
**Countdown:** derived from `deadline_unix_ms` minus a clock offset measured by `Ping`/`Pong`,
re-evaluated on `requestAnimationFrame`. Purely cosmetic — the server decides expiry.
**Reconnect:** exponential backoff (0.5s → 8s, jittered). On reconnect send `Hello` with the
stored `resume_token`; the server either restores the game or returns a typed error and the
UI falls back to the home screen with an explanation.
**Vietnamese input:** bind on `change`/submit and read `event.target.value`; never
re-write the input's value mid-composition, and listen for `compositionstart`/`compositionend`
before any normalization — rewriting the field while a Telex IME is composing corrupts
diacritic entry. Normalization is the server's job anyway.
**i18n:** every string lives in `lib/i18n/vi.js`, including the `RejectReason` map:
| Reason | Message |
|---|---|
| `TOO_FEW_SYLLABLES` | "Từ phải có ít nhất 2 tiếng." |
| `WRONG_LINK` | "Từ phải bắt đầu bằng tiếng \"{syllable}\"." |
| `NOT_IN_DICTIONARY` | "Không tìm thấy từ này trong từ điển." |
| `ALREADY_USED` | "Từ này đã được dùng rồi." |
| `NOT_YOUR_TURN` | "Chưa đến lượt bạn." |
| `TIMEOUT` | "Hết giờ!" |
**Theming:** CSS custom properties on `:root`, `[data-theme="dark"]` override, set from the
settings store before first paint to avoid a flash.
## Related Code Files
- Create: `web/` — SvelteKit scaffold (`package.json`, `svelte.config.js`, `vite.config.js`, `jsconfig.json`)
- Create: `web/src/lib/ws/client.js`, `web/src/lib/ws/messages.js`
- Create: `web/src/lib/stores/game.svelte.js`, `web/src/lib/stores/settings.svelte.js`
- Create: `web/src/lib/i18n/vi.js`
- Create: `web/src/lib/components/*.svelte` (per list above)
- Create: `web/src/routes/+layout.svelte`, `+page.svelte`, `play/+page.svelte`, `online/+page.svelte`
- Create: `web/src/app.css` — tokens, light/dark palettes
- Create: `web/src/lib/ws/client.test.js`, `web/src/lib/stores/game.test.js` (Vitest)
- Modify: `Makefile` — `make web`, `make web-dev`
- Modify: `server/internal/wsapi/server.go` — serve `NOITU_WEB_DIR` with SPA fallback
## Implementation Steps
1. Scaffold with `npx sv create web` (SvelteKit 2 / Svelte 5, JavaScript, Vitest, ESLint+Prettier); switch to `adapter-static` with `fallback: 'index.html'`.
2. Wire `make proto` output into `web/src/lib/proto`; add `@bufbuild/protobuf`.
3. `ws/client.js`: connect (`ws://` in dev via Vite proxy, same-origin `wss://` in prod), binary frames, decode to `ServerMessage`, dispatch to the store, ping/pong clock offset, backoff reconnect, `resume_token` in `sessionStorage`.
4. `stores/game.svelte.js`: `$state` model (phase, chain, currentSyllable, myTurn, deadline, scores, lastRejection, gameOver); one reducer per `ServerMessage` variant.
5. `stores/settings.svelte.js`: nickname, theme (system default, then explicit), `bestScore[difficulty]`, all `localStorage`-backed with try/catch so private-mode browsing still works.
6. Components: `ChainHistory` (scrolling word list, most recent pinned and highlighted, syllable-count badge on 3+ syllable words), `WordInput` (IME-safe, disabled when not your turn, autofocus on turn start), `CountdownRing` (SVG arc from the derived remaining time, colour shift under 5s), `NicknameInput` (20-char cap, mirrors the server rule), `ScoreBoard`, `DifficultyPicker`, `ConnectionBadge`, `GameOverPanel`, `ThemeToggle`, `AttributionFooter`.
7. Routes: home (nickname + mode + difficulty), `play` (the board, works for bot now and PvP in phase 7), `online` stub.
8. `i18n/vi.js` with every string and the rejection map; assert in a test that each proto `RejectReason` has an entry.
9. Personal best: on `GameOver`, compare and store per difficulty; show a "kỷ lục mới" marker.
10. `AttributionFooter`: names `minhqnd/dictionary`, links the repo and the CC BY-SA 4.0 deed — the user-visible half of the license obligation.
11. Vitest: store reducers for each server message, reconnect backoff schedule, rejection-map completeness, high-score persistence with `localStorage` throwing.
12. Verify the production bundle contains no word data (`grep` the built assets for known words) and record the bundle size.
## Success Criteria
- [ ] Full vs-bot game playable in a browser at all three difficulties — the protocol path is verified headlessly at all three; no browser playtest yet
- [ ] Words of 2, 3, and 4 syllables are all accepted and scored, with the syllable bonus visible
- [x] Nickname persists across reloads; the displayed name is always `accepted_nickname` from the server, never the raw input
- [x] Every rejection reason renders its specific Vietnamese message
- [ ] Countdown matches the server deadline within ~200ms; expiry is announced by the server, not the client
- [ ] Dark mode persists across reloads with no flash of the wrong theme
- [x] Personal best per difficulty persists and updates
- [ ] Killing the server mid-game shows a connection state and reconnects when it returns
- [ ] Diacritics typed with a Telex IME enter correctly on desktop and mobile
- [x] `npm run build` → `adapter-static` output served correctly by the Go binary, deep links included
- [x] Built assets contain no dictionary words
- [x] Attribution footer present with working links
- [x] `npm test` green
## Risk Assessment
| Risk | Signal | Response |
|---|---|---|
| IME composition broken by input handling | Diacritics mangled while typing | Never rewrite the input value during composition; guard with `compositionstart`/`end`. Explicitly tested on a real mobile keyboard, not just a desktop emulator |
| Countdown drift makes a legitimate move look late | Player sees 2s left, server says timeout | Clock offset from `Ping`/`Pong`; the ring visually settles ~300ms early so the client never claims more time than the server allows |
| Svelte 5 runes in `.svelte.js` store files misused | Reactivity silently stops updating | Stores use the documented `.svelte.js` rune-module pattern; store reducers are unit-tested outside components |
| Protobuf runtime bloats the bundle | Bundle budget exceeded | Measured in step 12; `protobuf-es` is tree-shakeable and only generated messages are imported |
| `localStorage` unavailable (private mode / blocked) | Crash on load | Every read/write wrapped in try/catch with a working in-memory default |
| Dev/prod WS URL divergence | Works in `npm run dev`, breaks when served by Go | Same-origin `wss://` resolved from `location` in prod; Vite proxy only in dev; both paths exercised before phase 7 |
## Phase 6 Outcome (2026-09-05)
The vs-bot UI is built, typed, tested, and served by the Go binary. 117 JavaScript tests pass
and `svelte-check` reports no errors across 368 files.
### What was verified, and how
| Claim | Evidence |
|---|---|
| The binary serves the SPA, deep links included | `curl` against a running server: `/`, `/play?difficulty=2` and `/online` all return `index.html`; hashed assets and the favicon return their own content types |
| Path traversal cannot escape the bundle | `/../server/go.mod` is normalised to a redirect; the percent-encoded form is refused with 400 |
| A bot game plays end to end at every difficulty | A headless client built from the same generated types played three moves against the bot at Easy, Medium and Hard against a fixture dictionary, with alternating turns and correct running scores |
| The nickname shown is the server's | The handshake returned `accepted_nickname` and the store adopts it |
| No wordlist reaches the browser | A bundle test greps every built file for dictionary words and asserts a size budget |
The bundle is 188,227 bytes across 28 files.
### Deliberate design points
**The store is a reducer over `ServerMessage` and nothing else.** Validity, turn order and the
result are read from the wire, never computed. That is what lets one screen serve both the bot
and, in phase 7, online play.
**The word field is uncontrolled.** A Telex or VNI input method composes a diacritic across
several keystrokes, and writing the value back on each keystroke cancels the composition. The
field is read on submit and cleared only there, where composition has already ended.
**The countdown settles 300ms early.** A ring showing time left while the server has already
timed the player out reads as the game cheating; reaching zero a fraction early reads as
rounding.
**`Hello` is sent before the open status is announced.** The server refuses everything before
the handshake, so a listener reacting to "connected" by sending a message would otherwise race
it. Ordering it explicitly turns a timing accident into a contract.
### Defects found by review
Review found one blocking defect and four user-visible breakages, all in code no test was
driving. The pattern behind the worst of them is worth naming: **the game lifecycle was
inferred from the game model instead of being owned.**
**Every rematch started two bot games, and the second killed the first.** The screen sent
`StartBotGame` from an effect whenever the board was idle. Clearing the board for a rematch is
exactly that condition, so the rematch button sent one request and the inference sent another.
The server allocates a room per request and does not deduplicate, so the player got two rooms
sharing one socket: the abandoned room kept its turn timer, overwrote the turn sequence that
the live game's submissions were tagged with, and about one turn later ended the game in a loss
nobody could explain. A request is now stored intent — `web/src/lib/stores/bot-session.svelte.js`
— and clearing the board cannot mean anything. Twelve tests cover it, because the absence of a
test over this wiring is why it shipped.
**Arriving at the board with a finished game on screen never started a new one.** Same root
cause: nothing reset the store on entry, so the header link or the browser Back button left a
stale phase behind and the idle condition never held. The screen now owns the socket and the
game for as long as it is mounted, and gives the current game up on the way out rather than
leaving a room whose timer keeps running somewhere the player cannot see.
**A word typed during a reconnect vanished silently.** The input stayed enabled while the socket
was down, the send returned false, and the field cleared anyway. The field is now gated on the
connection as well as the turn, and only clears a word that actually went out.
**A protocol bump would have put every open tab into a reconnect loop.** The backoff reset when
the socket opened, but the server accepts the connection and *then* refuses an unspeakable
`Hello`, so every retry looked like a success and restarted from the shortest delay. The
backoff now resets on `Welcome`, and `protocol_version_mismatch` stops reconnection outright.
**The opening turn counted down against the raw device clock.** The first clock probe was
scheduled a full interval out, but `GameStarted` arrives one round trip after `Hello`. A device
clock behind the server would have shown more time than the server allowed — the exact thing
the settle margin exists to prevent. The probe now goes out with the handshake.
Also fixed from the review: cache headers on the static bundle, where an `index.html` cached
across a deploy names hashed assets that no longer exist and the app loads into a blank page;
a record-keeping effect that read and wrote the same reactive state; an animation loop that ran
at 60fps between turns; and the difficulty picker, which claimed a radio role without radio
keyboard behaviour and now uses real radio inputs.
### Defects found while building
**The bundle guard was vacuous on its first negative test.** Planting an unused dictionary word
in the string table did not fail it, because Rollup dropped the unused property. Re-testing with
a rendered string made it fail correctly. It now also decodes `\uXXXX` escapes, so a minifier
configured to emit ASCII cannot hide the words from a substring search.
**`jsdom` broke the phase-4 fixture test.** Making it the global test environment rewrote
`import.meta.url` to an `http:` URL, so the cross-language fixture reader could not resolve its
path. The DOM is now opted into per file by the two suites that need it.
**`svelte-check` found a required prop that no caller passed.** `DifficultyPicker` declared
`onselect` as required while the home screen only binds `value`. The prop is optional now.
### Guards added after review
| Guard | What it would catch |
|---|---|
| `tests/bot-session.test.js` | A rematch sending two requests, or a result scored twice |
| `tests/error-codes.test.js` | A server error code with no Vietnamese message, read from the Go source that emits them; it also fails on copy for a code the server cannot send |
| `TestStaticCacheHeaders` | The shell becoming cacheable across a deploy |
| Build freshness in `tests/bundle.test.js` | The wordlist check reading a stale bundle |
| Handshake ordering in `tests/ws-client.test.js` | `Hello` losing its place ahead of the open status |
The error-code guard and the wordlist guard were both negative-tested by breaking what they
protect and watching them fail for the right reason. The bundle budget was 1.5 MB against a
188 KB bundle, which caught nothing; it is 400 KB now.
### Deviations from the plan
- Scaffolded by writing the config files directly rather than `npx sv create`, which is
interactive. ESLint and Prettier were not added; `svelte-check` is wired into `make test-web`
instead.
- Tests live in `web/tests/`, following the layout phase 4 established, not beside their
sources.
- `npm test` now runs `vite build` first. The bundle check reads built output, and a stale
build would let it pass over code that no longer exists. This also means the proto workflow's
JavaScript step builds the frontend, which needs no dictionary and stays fast.
### Not verified
Five criteria need a real browser and are left unchecked: one-handed mobile use, Telex
diacritic entry, the absence of a theme flash, reconnect after the server is killed mid-game,
and the rematch button. The logic behind the last three is unit-tested — the pre-paint theme
script, the jittered backoff schedule, and the game-request latch — but none has been watched
happen. The rematch path is worth naming separately: it is where review found the blocking
defect, and the fix is verified by unit tests over the latch rather than by playing a game.
The countdown criterion is no longer known to be violated on the opening turn, since the clock
probe now goes out with the handshake, but the ~200ms figure has not been measured.
A resumed session shows the opening word and the last move rather than the whole chain. That
is the server's replay contract, which rebuilds the position from the engine rather than
replaying a recorded stream; the client renders what it is sent.
plans/260904-1125-noi-tu-web-game/phase-07-online-1v1-and-release.md
-277
View File
@@ -1,277 +0,0 @@
---
title: "Phase 7: Online 1v1 and Release"
status: done
phase: 7
priority: P1
effort: "4d"
dependencies: [5, 6]
---
# Phase 7: Online 1v1 and Release
## Overview
Complete the online 1v1 experience end to end in the browser — room codes, waiting room,
opponent presence, reconnect UX, rematch — then verify the whole game with browser E2E tests
and ship it: CI, container image, deployment, and documentation.
## Requirements
**Functional**
- [x] Create a room → shareable 6-character code and a copyable invite link
- [x] Join by code, or by opening an invite link with the code prefilled
- [x] Waiting room until the opponent arrives; leaving cleans the room up
- [x] Both players' nicknames shown in the waiting room and scoreboard (server-sanitized values only)
- [x] Live opponent state: their turn, their timer, their disconnect and reconnect
- [x] Resign, and rematch in the same room after a game ends
- [x] Clear Vietnamese error states: room not found, room full — a room whose game has started is full, so there is no third state
**Non-functional**
- [x] E2E coverage of both modes with two real browser contexts
- [ ] CI: Go tests + race, JS tests, `buf lint`/`breaking`, generated-code drift check, builds
- [x] Deployable artifact: container image with the binary, the static frontend, and `noitu.db`
- [x] README documents setup, the license split, and deployment
## Architecture
No new server concepts — phase 5 already implements rooms, codes, reconnect, and resign.
This phase is the frontend surface for them plus release engineering.
**Invite link:** `https://<host>/online?code=ABC123`. The `online` route reads the query
param, prefills, and auto-joins when the code is well-formed. Room codes use an unambiguous
alphabet (no `0/O/1/I`) and are displayed grouped for reading aloud.
**Online screen states**
```
idle ──create──► waiting (code shown, copy button)
│ └──opponent joins──► playing
└──join(code)──► playing | error(room_not_found | room_full | already_started)
playing ──opponent disconnects──► banner "Đối thủ mất kết nối… (Ns)"
├─ they return ──► resume playing
└─ grace expires ──► GameOver(END_OPPONENT_LEFT)
```
**Rematch:** after `GameOver` in a PvP room, either player may request a rematch; the room
resets its engine with a fresh opening word when both accept, or closes on decline/timeout.
Requires a small additive protocol change — `RequestRematch` / `RematchState` — which is why
it lands here rather than in phase 4: additive tags only, per the phase-4 compatibility rule.
**Deployment — Docker image on a VPS behind a reverse proxy (validated decision):**
```
Dockerfile (multi-stage)
node → npm ci && npm run build → /web/build
go → CGO_ENABLED=0 go build ./cmd/... → /noitu-server
data → curl the 179 MB dictionary.db → build-dictionary → /data/noitu.db
final → distroless/static + binary + /web + /data/noitu.db + /data/LICENSE + ATTRIBUTION
```
`CGO_ENABLED=0` works because the SQLite driver is `modernc.org/sqlite` (pure Go) — the
payoff for that phase-1 decision. The 179 MB upstream download happens **only in the data
builder stage**, so it never reaches the final image; `noitu.db` is copied in as its own
layer, keeping the CC BY-SA 4.0 artifact physically distinct from the Apache-2.0 binary.
**CI data strategy (consequence of not republishing a derived DB):** CI must never download
179 MB per run.
| Job | Data used |
|---|---|
| Unit tests (every push/PR) | Small fixture DB built in-test from a checked-in word sample (phase 2 helper) |
| E2E tests (every push/PR) | Same fixture DB, ~200 curated words — enough to script deterministic games |
| Full dictionary build (manual / nightly) | Real upstream, with the download cached by release tag |
| Docker image build (release only) | Real upstream, inside the builder stage |
## Related Code Files
- Modify: `proto/noitu/v1/game.proto` — add `RequestRematch`, `RematchState` (new tags only)
- Modify: `server/gen/...`, `web/src/lib/proto/...` — regenerate
- Modify: `server/internal/wsapi/room.go` — rematch handling, room reset
- Modify: `web/src/routes/online/+page.svelte` — replace the phase-6 stub with the full create/join/waiting/error flow
- Create: `web/src/lib/components/RoomCodePanel.svelte`, `WaitingRoom.svelte`, `OpponentStatus.svelte`, `RematchPrompt.svelte`
- Modify: `web/src/lib/stores/game.svelte.js` — room/opponent/rematch state
- Modify: `web/src/lib/i18n/vi.js` — online-mode strings
- Create: `e2e/bot-game.spec.js`, `e2e/pvp-game.spec.js`, `e2e/reconnect.spec.js` (Playwright)
- Create: `playwright.config.js`
- Create: `Dockerfile`, `.dockerignore`
- Create: `.github/workflows/ci.yml`
- Modify: `README.md` — quickstart, architecture, deployment, license split
- Create: `docs/deployment.md`
## Implementation Steps
1. Add `RequestRematch` / `RematchState` to the proto with new tags; regenerate both targets; extend the round-trip tests.
2. Server: rematch in `room.go` — both-accept resets the engine with a new opening word and re-emits `GameStarted`; decline or timeout closes the room. Cover with a room test.
3. `online/+page.svelte`: the state machine above, code input (uppercase, alphabet-restricted, paste-friendly), `?code=` prefill and auto-join.
4. `RoomCodePanel`: large grouped code, copy button, copy invite link, Web Share on mobile where available.
5. `WaitingRoom`: opponent-pending state with a cancel that leaves the room cleanly.
6. `OpponentStatus`: turn indicator, disconnect banner with the grace countdown, reconnect confirmation.
7. `RematchPrompt` on the game-over panel for PvP; bot mode keeps a plain "chơi lại".
8. Vietnamese strings for every new state, including the three join errors.
9. Playwright setup: build the frontend, run the real server against the **~200-word fixture DB** (never the real dictionary — CI must not download 179 MB), run specs against it.
10. E2E `bot-game.spec.js`: pick Hard, play scripted valid and invalid words, assert rejection messages, play to game over, assert the score and personal-best marker.
11. E2E `pvp-game.spec.js`: two browser contexts with different nicknames, create + join by code, assert each side shows the other's sanitized nickname, alternate turns, assert each side sees the other's word, resign ends it correctly.
12. E2E `reconnect.spec.js`: drop one context's socket mid-game, assert the opponent's disconnect banner, restore within grace, assert the board matches on both sides.
13. `Dockerfile` multi-stage as above; verify the image runs with only `NOITU_DB_PATH` set.
14. `ci.yml`: `go vet`, `go test ./... -race`, `npm test`, `npm run build`, `buf lint`, `buf breaking`, generated-code drift check, Playwright — all on the fixture DB. Separate release-only job builds the Docker image. Add an assertion that the built image contains `data/LICENSE` and `data/ATTRIBUTION.md`.
15. `README.md`: what the game is, quickstart (**`make fetch-dict` downloads 179 MB once**, then `make dict`), `make` targets, architecture diagram, **License** section (Apache-2.0 code / CC BY-SA 4.0 data with attribution), and a link to `data/ATTRIBUTION.md`.
16. `docs/deployment.md`: env vars (incl. `NOITU_TURN_LIMIT=20s`, `NOITU_ALLOWED_ORIGINS`), TLS/`wss://` behind a reverse proxy (`Upgrade`/`Connection` headers, proxy read timeout longer than the WS keepalive, buffering disabled), health check, and how `noitu.db` reaches the container.
## Success Criteria
- [x] Two independent browser contexts play a full game via a shared room code; two physical machines untested
- [x] Invite link opens straight into the room
- [x] Both join errors the protocol has show correct Vietnamese messages, plus a malformed code caught before it is sent
- [x] Disconnect shows the opponent a grace countdown; return inside it resumes with identical boards; expiry awards the win
- [x] Rematch restarts in the same room with a new opening word
- [x] Resign ends the game immediately with the right winner
- [x] Playwright suite green: bot game, PvP game, reconnect
- [ ] CI green on all steps including the generated-code drift check, without ever downloading the 179 MB upstream DB
- [x] `docker run` serves a playable game; the image contains `data/LICENSE`, `data/ATTRIBUTION.md` and `NOTICE` but no upstream file — verified against a fixture-dictionary build, not an upstream one
- [x] README license section and in-app attribution agree with `data/ATTRIBUTION.md`
- [ ] Full pass over `plan.md` success criteria — every box checkable
## Risk Assessment
| Risk | Signal | Response |
|---|---|---|
| Reverse proxy breaks WS upgrade in production | Works locally, 400/502 on deploy | `docs/deployment.md` states the required proxy headers and timeouts; the health check plus a post-deploy WS smoke test catch it before users do |
| Playwright PvP tests flake on timing | Intermittent CI red | Drive by deterministic server events (wait for the turn indicator), never fixed sleeps; run the suite with a long turn limit via env |
| Rematch protocol addition breaks phase-6 clients | Decode errors in an already-running client | Additive tags only; `buf breaking` in CI is the guard, and `protocol_version` lets the server reject a stale client cleanly |
| Room codes guessable enough to join a stranger's game | Reports of uninvited joins | `crypto/rand` codes plus per-IP join rate limiting from phase 5; 32^6 space with rate limiting makes scanning impractical |
| CC BY-SA obligations lost during packaging | Image ships without `data/LICENSE` | The Docker copy step includes the whole `data/` directory, and a CI assertion checks `data/LICENSE` and `data/ATTRIBUTION.md` exist in the built image |
| Scope creep into accounts/leaderboards at the finish line | New requests during release work | Explicit non-goals in `plan.md`; log them as post-v1 items instead |
## Phase 7 Outcome (2026-09-05)
Online 1v1 is playable end to end, the whole game is covered by a browser suite, and the
result ships as one 24 MB container image.
### What was verified, and how
| Claim | Evidence |
|---|---|
| Two players join by code and alternate turns | Two browser contexts against the real binary; each side sees the other's sanitized nickname and the other's words |
| An invite link opens straight into the room | The second context only opens a URL — no code typed, no button pressed |
| A rematch restarts the same room | One acceptance shows the other player a prompt; the second starts a new game with a fresh opening word and the room code unchanged |
| A disconnect is announced and survivable | Taking a player's page away raises the opponent's banner; returning inside the window restores the same position and clears it |
| An opponent who never returns forfeits | The grace window expires and the win is awarded, with the reason shown |
| The image runs and carries its obligations | `docker run` serves the app and a deep link; the image contains `data/LICENSE`, `data/ATTRIBUTION.md` and `NOTICE`, and no upstream database |
| The difficulty ladder is real | The simulation suite from phase 3 reports Hard beating Easy 94 games to 6, Medium beating Easy 93 to 7, and Hard beating Medium 78 to 22 |
21 browser tests, 139 JavaScript unit tests, and the Go suite under `-race` all pass.
### Deliberate design points
**The room now outlives its game, but only when there is someone to ask.** A finished game
opens a rematch offer when both seats still hold live connections. A bot room closes exactly
as before, because there is nothing to negotiate with a bot and keeping it alive would leak a
goroutine and an engine per finished game.
**There is no decline message.** Leaving is the decline, and the server already learns about
that from the socket closing. One message and one timeout cover every way a rematch does not
happen, which is a smaller protocol and one less state to get wrong.
**`turn_seq` no longer restarts at one.** A rematch reuses the same connections, so a
submission still in flight from the previous game could otherwise match a turn in the new one
and be applied to it.
**The fixture dictionary goes through the real builder.** `build-dictionary --words` reads a
checked-in list and runs it through the same filter, alias and write path production uses, so
a test database cannot drift into being shaped differently from what the server loads. Its
graph has exactly one syllable productive enough to open on, which is what makes a scripted
game deterministic without pinning the bot's replies.
### Defects found by building the browser suite
**A player who refreshed mid-game could not get back in.** The online screen waited for the
player to ask for a room before connecting — correct for someone who has just arrived and is
still typing their name, wrong for a tab that already holds a session. It now reconnects
immediately when a resume token is present.
**The opponent's disconnect banner never cleared.** The server restored the seat but told
nobody, so the other player watched "đối thủ mất kết nối" for someone already playing again.
A resume now announces itself to the opponent.
**A dead socket was invisible to the client.** Nothing noticed a connection that failed
without closing, so the page kept showing a live connection and a running countdown over a
socket nothing could reach. The client now treats silence longer than three ping intervals as
a dead socket and reconnects.
**The nickname typed on the online screen never reached the server.** The socket opened on
arrival and `Hello` carries the name once, so both players were introduced under whatever was
stored before they got there. Connecting when the player actually asks for a room fixed it.
### Two tools that did not work, and why
**Offline emulation does not cut a loopback socket.** A test that "went offline" kept playing
happily against the local server. Routing the WebSocket cuts it for real.
**A routed socket's close does not reliably reach the server.** The page observes it, but the
grace window never starts. That makes socket routing the right tool for what the client does
about a dead connection and the wrong one for what the server does about a missing player; the
latter tests take the page away instead. Both limitations are recorded in `e2e/socket-cut.js`
so the next person does not rediscover them.
### Deviations from the plan
- Playwright lives under `web/` rather than the repository root, so there is still one npm
package rather than two.
- There are two join errors, not three: a room whose game has started is full, and the server
has no separate code for it. A malformed code is caught in the client before it is sent.
- CI is split across two workflows. `proto.yml` keeps the wire contract, `ci.yml` owns the
tests and the image, and the duplicated test steps were removed from the former.
- The image is built on every push against the fixture word list, not only on release. An
image built only at release time is an image that breaks at release time, and the build
argument that makes this cheap already existed for local testing.
### Defects found by review
Review reproduced three defects, two of them consequences of the new state this phase
introduced: a room that outlives its game.
**A game that ended on the turn clock never offered a rematch, and the button was still
there.** The offer was opened from the message arm of the room loop, so the most common
natural ending — running out of time — closed the room with nothing to accept. The player
pressed "chơi lại" and got an error. The end-of-game decision now sits after the whole select,
so every way a game can end reaches it, and the online game-over panel no longer shows a
rematch button at all: the prompt appears only while an offer is actually open, so it is the
only thing that can ask.
**A second connection presenting the same resume token disconnected the player who was still
there.** `resumeFrom` retired the old connection as soon as the room had *accepted the
message*, but the room decides asynchronously and refuses a resume into a finished game — which
is exactly the state during a rematch offer. A duplicated tab therefore ended the room. The
room now retires the old connection only once it has agreed to the swap. Before this phase the
window did not exist, so a refused resume was unreachable.
**Rooms leaked when one connection created several.** `attach` overwrote the session's room
pointer and nothing told the old room, which then parked in `select` forever holding a
goroutine and a room code. Rate limiting bounds the rate, not the total, and connections are
free, so this was unbounded on a public endpoint. `attach` now releases the room it is leaving.
This one predates the phase; it is fixed here because this is the phase that puts the server on
the internet.
Also fixed: `RequestRematch` had no rate limit despite being the only client message that fans
out to both players, so a burst could fill the opponent's outbox until the server closed their
session. A stale disconnect notice was being treated as a rematch decline, which needed
`handleDisconnect` to stop conflating "this notice applied" with "a game is still running".
The client's liveness check could fire on a throttled background tab and close a healthy
socket, so it now ignores a tick that was itself late. And a stale resume token opened the
lobby with an error the player did nothing to cause.
Each of the three defects has a regression test, and the two that were reachable through the
room loop were negative-tested by reverting the fix and watching them fail.
### Not verified
CI has not run on GitHub. Every step passes locally, including the image build and its licence
assertions, but the workflows themselves are unexercised.
The image was built from the fixture word list, not the 179 MB upstream release. The
downloading branch of the `Dockerfile` is the same shape, pinned by the same checksum the
`Makefile` uses, but it has not been run here.
Two players on two physical machines have not played. Two independent browser contexts have,
which exercises everything except the network between them.
plans/260904-1125-noi-tu-web-game/plan.md
-381
View File
@@ -1,381 +0,0 @@
---
title: "Noi Tu Web Game"
description: "Vietnamese nối từ web game — SvelteKit frontend, Go backend, WebSocket + Protobuf, server-authoritative dictionary over SQLite. Vs-bot and online 1v1."
status: complete
priority: P1
effort: "~3-4w"
tags: [game, sveltekit, go, websocket, protobuf, sqlite, vietnamese]
created: 2026-09-04
blockedBy: []
blocks: []
---
# Noi Tu Web Game
## Overview
Web implementation of **nối từ**, the Vietnamese word-chain game: a player submits a
meaningful word of **at least 2 syllables** whose **first syllable equals the previous
word's last syllable**. No reuse, timed turns. Loss on timeout, invalid word, wrong link,
repeat, or no legal move remaining.
Two modes ship in v1: **vs bot** (3 difficulties) and **online 1v1**. Both run through the
same server-authoritative engine — the browser never holds the wordlist, so validation
cannot be bypassed and the bot and PvP paths share one code path.
**Stack (user-selected):** SvelteKit (JavaScript) frontend · Go backend · WebSocket
transport with Protobuf framing · dictionary served from SQLite server-side.
**Research basis:** [`plans/reports/research-260904-1058-noi-tu-game.md`](../reports/research-260904-1058-noi-tu-game.md)
## Goals
| # | Goal | Priority |
|---|------|----------|
| 1 | Correct, server-authoritative nối từ rules incl. Vietnamese text normalization | P1 |
| 2 | Reproducible dictionary pipeline from `minhqnd/dictionary` with correct CC BY-SA 4.0 compliance | P1 |
| 3 | Typed WS protocol (Protobuf) shared by Go and JS, single source of truth | P1 |
| 4 | Playable vs-bot mode with 3 difficulty levels | P1 |
| 5 | Online 1v1 with nicknames, room codes, turn timer, reconnect | P1 |
| 6 | Vietnamese UI, dark mode, local high score | P2 |
| 7 | Single deployable Go binary, shipped as a Docker image | P2 |
## Non-goals (v1)
- Word definitions / meanings display (explicitly out of scope per user selection)
- Accounts, auth, persistent profiles, server-side leaderboards
- 4-player *đấu trường* mode
- Native mobile apps
- Player-submitted word additions / dictionary moderation UI
- Redistributing our derived dictionary as a downloadable artifact (see Data Distribution)
## Architecture
```
┌──────────────────────────────┐ ┌────────────────────────────────────┐
│ SvelteKit SPA (adapter- │ WSS │ Go server (single binary) │
│ static, Svelte 5 runes) │ ◄─────► │ │
│ │ Protobuf│ wsapi ── hub ── room(s) │
│ lib/ws connection+codec │ binary │ │ │
│ lib/proto protobuf-es gen │ frames │ ├── game.Engine │
│ lib/stores game state │ │ └── bot.Bot │
│ routes/ UI (vi, dark mode)│ │ │ │
└──────────────────────────────┘ │ dictionary.Store ──┘ │
localStorage: nickname, high score, │ │ read-only SQL │
theme │ data/noitu.db (CC BY-SA 4.0) │
└────────────────────────────────────┘
▲ built locally by
server/cmd/build-dictionary (Go)
▲ reads
dictionary.db (179 MB, downloaded from
minhqnd/dictionary release v2.0.0)
```
**Key decisions**
| Decision | Choice | Rationale |
|---|---|---|
| Authority | Server validates every move; client is a view | User chose server-side dictionary; also the only way PvP is cheat-resistant |
| Word length | **≥ 2 syllables**, link on first↔last syllable | User decision. Closer to casual play; keeps 3+ syllable compounds in the corpus |
| Bot location | Server-side, same engine as PvP | One rule implementation, no duplication (DRY); bot "thinking delay" is server-timed |
| Turn limit | **20s**, identical for bot and PvP | One constant, one code path; `NOITU_TURN_LIMIT` overrides |
| Player identity | **User-typed nickname**, no accounts | User decision. Stored in `localStorage`, sanitized server-side |
| WS library (Go) | `github.com/coder/websocket` v1.8.15 (ISC) | `gorilla/websocket` archived 2022 and panics on concurrent writes; coder/websocket is context-aware and handles concurrent writes |
| Protobuf (JS) | `@bufbuild/protobuf` v2.14.1 + `protoc-gen-es` | Only fully conformant JS impl, ESM/tree-shakeable, small browser bundle |
| Protobuf (Go) | `protoc-gen-go` via same `buf.gen.yaml` | One schema, two targets, generated in CI |
| Go module path | `github.com/tiennm99dev/noitu/server` | Matches the actual git remote |
| SQLite driver | `modernc.org/sqlite` (CGo-free) | Keeps `CGO_ENABLED=0` cross-compilation and a distroless image; read-only lookups are not CGo-bound |
| Dictionary DB | Loaded at runtime from `data/noitu.db`, **not** `go:embed` | Keeps CC BY-SA 4.0 data a separate artifact from Apache-2.0 code — cleanest license boundary |
| Tone variants | Alias table built offline (`hoà`→`hòa`) | Precomputed lookup beats a fragile runtime tone-placement algorithm (KISS) |
| Room state | In-memory, no DB | v1 has no persistence requirement; restart drops live games (accepted, documented) |
| Deployment | Docker image on a VPS behind a reverse proxy | Most portable; phase 7 ships a distroless image and proxy docs |
## Data Distribution
**Decision (validated):** we do **not** republish a derived dictionary. Every build downloads
`dictionary.db` from the upstream release and derives `data/noitu.db` locally.
Upstream asset: [`minhqnd/dictionary` release v2.0.0 → `dictionary.db`](https://github.com/minhqnd/dictionary/releases/download/v2.0.0/dictionary.db) — **179 MB**.
Consequences, all handled explicitly:
| Consequence | Handling |
|---|---|
| 179 MB is too large to fetch on every CI run | CI runs tests against a **small fixture DB** built from a checked-in word sample (phase 2). Only a manually-triggered/nightly job downloads the real upstream and builds the full DB |
| Docker build needs the upstream DB | Downloaded in the **builder stage** only; the final image carries just the derived `noitu.db`, so the 179 MB never ships |
| `data/noitu.db` is a local build artifact | Git-ignored. `make dict` is a documented prerequisite for running the real server |
| The derived DB *is* distributed inside our Docker image | CC BY-SA 4.0 still applies to that image layer — attribution files ship with it, asserted in CI (phase 7) |
## Licensing (mandatory — CC BY-SA 4.0 share-alike)
`minhqnd/dictionary` is **dual-licensed**: MIT for application code, **CC BY-SA 4.0 for the
dictionary data**. We consume only the data, so share-alike applies to our derived wordlist
wherever we distribute it (notably the Docker image).
| Artifact | License | File |
|---|---|---|
| All source code in this repo | Apache-2.0 (existing) | `LICENSE` |
| Derived dictionary `data/noitu.db` and the build script's output | **CC BY-SA 4.0** | `data/LICENSE` |
| Attribution + list of modifications | — | `data/ATTRIBUTION.md` |
| Root pointer to the split | — | `NOTICE`, README section |
| User-visible credit | — | in-app footer with link |
Attribution must name: `minhqnd/dictionary`, its own upstream sources (Wiktionary, `vntk/dictionary`),
the CC BY-SA 4.0 license URL, and **what we changed** (filtered to Vietnamese entries of ≥2
syllables, NFC-normalized, tone-variant aliases added, first/last syllable columns and indexes
added, all non-`vi` languages and all definitions/translations dropped).
## Phases
| # | Phase | Status | Depends on |
|---|-------|--------|-----------|
| 1 | [Foundations and Data Pipeline](./phase-01-foundations-and-data-pipeline.md) | Complete | — |
| 2 | [Go Dictionary and Normalization](./phase-02-go-dictionary-and-normalization.md) | Complete | 1 |
| 3 | [Go Game Engine and Bot AI](./phase-03-go-game-engine-and-bot-ai.md) | Complete | 2 |
| 4 | [Protobuf Contract and Codegen](./phase-04-protobuf-contract-and-codegen.md) | Complete | 1 |
| 5 | [Go WebSocket Server and Rooms](./phase-05-go-websocket-server-and-rooms.md) | Complete | 3, 4 |
| 6 | [SvelteKit Frontend](./phase-06-sveltekit-frontend.md) | Complete | 4, 5 |
| 7 | [Online 1v1 and Release](./phase-07-online-1v1-and-release.md) | Complete | 5, 6 |
Phases 2-3 and 4 are independent after phase 1 and can run in parallel if desired.
## Target Repository Layout
```
LICENSE # Apache-2.0 — code only
NOTICE # points at data/ licensing
README.md
buf.yaml buf.gen.yaml
proto/noitu/v1/game.proto # single protocol source of truth
data/
LICENSE # CC BY-SA 4.0 full text
ATTRIBUTION.md
dictionary.db # upstream, 179 MB, git-ignored, downloaded
noitu.db # derived build artifact, git-ignored
server/cmd/build-dictionary/ # Go: dictionary.db -> noitu.db
server/ # module github.com/tiennm99dev/noitu/server
go.mod
cmd/noitu-server/main.go
internal/vietnamese/ # normalize.go, syllable.go
internal/dictionary/ # store.go — read-only SQLite lookups
internal/game/ # engine.go, state.go, rules.go
internal/bot/ # bot.go, strategy_*.go
internal/wsapi/ # hub.go, room.go, session.go, codec.go
gen/noitu/v1/ # protoc-gen-go output
web/
src/lib/proto/ # protobuf-es output
src/lib/ws/ # socket client, reconnect, codec
src/lib/stores/ # game state (Svelte 5 runes)
src/lib/components/
src/routes/
```
## Success Criteria
- [x] `server/cmd/build-dictionary` reproducibly turns the upstream `dictionary.db` into `data/noitu.db`; entry count and ≥2-syllable purity asserted
- [x] `data/LICENSE`, `data/ATTRIBUTION.md`, `NOTICE`, README license section, and in-app credit all present and consistent
- [x] Go engine unit tests cover: wrong link, unknown word, reuse, single-syllable input, timeout, no-legal-move, and the tone-variant cases `hoà/hòa`, `thuý/thúy`, `quí/quý`
- [x] Words of 2, 3, and 4 syllables all accepted and chain correctly on first↔last syllable
- [x] One `.proto` generates working Go and JS clients; no hand-written message types
- [x] Vs-bot playable end to end at all 3 difficulties; Hard bot wins measurably more than Easy over 100 simulated games
- [x] Online 1v1: two browsers join by room code with chosen nicknames, alternate turns, server-enforced 20s timer, correct win/loss, reconnect within grace window restores the game
- [x] Vietnamese UI throughout; nickname and high score persist, checked in a real browser. The dark-mode toggle persists by the same mechanism but has only been unit-tested
- [x] `go build` produces one binary; `web` builds to static assets served by that binary
- [x] No wordlist reachable from the client bundle (verified by inspecting the built assets)
- [ ] CI runs green without ever downloading the 179 MB upstream DB — the workflows are written and every step passes locally, but they have not run on GitHub
## Risk Assessment
| Risk | Signal it happened | Response |
|---|---|---|
| Upstream `dictionary.db` yields too few clean ≥2-syllable entries (<40k) | Build script count assertion fails | Union with Viet74K multi-syllable filter (see research report §5) — decided before implementation, not mid-phase |
| Tone-variant aliasing misses real-world spellings | Players report valid words rejected | Alias table is data, not code: regenerate with an expanded rule set and rebuild `noitu.db`; add a rejected-word log to find gaps |
| CC BY-SA share-alike misapplied to code | License review flags the Apache/CC mix | Data stays a runtime-loaded separate artifact, never `go:embed`-linked; boundary documented in `NOTICE` and asserted in the image |
| Hard bot is unbeatable, players quit | Playtest win rate ≈0% vs Hard | Cap Hard's killer-syllable play rate; the difficulty ladder is tunable constants, not structure |
| In-memory rooms lost on deploy/restart | Live games drop | Accepted for v1 and stated in-app ("máy chủ đang khởi động lại"); persistence is a post-v1 item |
| Protobuf schema churn breaks a deployed client | Client decode errors after deploy | Additive-only field changes, reserved tags, version field in the handshake; server rejects unknown protocol versions with a clear message |
| Nicknames become an abuse surface (shown to strangers) | Offensive names reported | Server-side sanitization: length cap, control-char strip, whitespace collapse, empty → auto-name. A denylist is a cheap post-v1 addition if needed |
| Allowing 3+ syllable words admits phrases that feel wrong | Playtest complaints about unnatural entries | Builder supports a max-syllable cap and a denylist file; both are data changes, no code change |
## Validation Log
### Session 1 — 2026-09-04
**Verification Results**
- Tier: Full (7 phases)
- Claims checked: 14 · Verified: 11 · Failed: 1 · Imprecise: 2 · Unverified (env): 1
| Claim | Result | Evidence |
|---|---|---|
| proto `go_package` = `github.com/tiennm99/noitu/server/...` | **FAILED** | `git remote -v` → `github.com/tiennm99dev/noitu.git`. Corrected to `github.com/tiennm99dev/noitu/server` |
| `coder/websocket` provides `SetReadLimit`, `AcceptOptions.OriginPatterns`, `MessageBinary` | VERIFIED | pkg.go.dev, v1.8.15, ISC |
| "Read deadline per frame" via a socket deadline setter | **IMPRECISE** | coder/websocket exposes no `SetReadDeadline`; deadlines are `context.WithTimeout` on `Read`. Phase 5 corrected |
| `dictionary.db` downloadable from `minhqnd/dictionary` Releases | VERIFIED | GitHub API: release `v2.0.0`, asset `dictionary.db` |
| Upstream DB size assumed manageable | **IMPRECISE** | Asset is **179 MB**. Plan had no size claim; Data Distribution section added |
| `@bufbuild/protobuf` / `@bufbuild/protoc-gen-es` current | VERIFIED | npm registry, both v2.14.1 |
| Go ≥1.24, Node, Docker available locally | VERIFIED | go1.26.5, node v24.18.0, Docker 29.6.2 |
| `buf` installed locally | **UNVERIFIED (env gap)** | `buf: command not found`. Not blocking — generated code is committed, so `buf` is needed only to change the schema |
**Decisions confirmed**
| # | Question | Decision | Impact |
|---|---|---|---|
| 1 | Go module path | `github.com/tiennm99dev/noitu/server` | Phase 4 `go_package` corrected |
| 2 | Derived DB distribution | Do not republish — every build downloads upstream and derives locally | New Data Distribution section; phase 1 download step, phase 7 CI + Docker strategy |
| 3 | Turn time limit | 20s, identical for bot and PvP | Phase 5 default constant |
| 4 | Deployment target | Docker on a VPS behind a reverse proxy | Phase 7 docs scoped to proxy config, not a PaaS |
| 5 | Word length rule | **≥ 2 syllables** (changed from strict 2) | Phases 1, 2, 3, 4, 6 — filter, API, reject reason, proto enum, UI copy |
| 6 | Player identity in PvP | **User-typed nickname** | Phases 4, 5, 6, 7 — proto field, sanitization, input UI, display |
**Phase propagation:** phases 1-7 all updated. Reject reason renamed
`NOT_TWO_SYLLABLES` → `TOO_FEW_SYLLABLES` across the engine, proto, and Vietnamese copy.
### Whole-Plan Consistency Sweep
Re-read `plan.md` + all 7 phase files after propagation.
| Check | Result |
|---|---|
| Stale "2-syllable"/"exactly 2" claims | Resolved — all now "≥2 syllables" / "at least 2" |
| `NOT_TWO_SYLLABLES` enum name | Resolved — renamed everywhere (engine, proto, i18n) |
| `go_package` module path | Resolved — single correct value in phase 4 |
| Turn limit stated inconsistently | Resolved — 20s in plan.md + phase 5, `NOITU_TURN_LIMIT` override |
| Nickname referenced but never sourced | Resolved — proto field, sanitization, input, and display now specified |
| `online/+page.svelte` Create vs Modify across phases 6/7 | Resolved — phase 6 creates the stub, phase 7 modifies it |
| Data distribution vs Docker/CI steps | Resolved — consistent across plan.md, phase 1, phase 7 |
| Duplicate embedded proto schema | Single copy, in phase 4 only |
**Unresolved contradictions: none.**
## Phase 1 Outcome (2026-09-04)
Built and verified against the real upstream release.
| Metric | Value |
|---|---|
| Words | 48,216 (floor 40,000) |
| 2 / 3 / 4+ syllables | ~40k / ~4.5k / ~3.5k |
| Aliases | 2,041 |
| Derived DB | ~3 MB, from a 179 MB source |
| Coverage | `internal/vietnamese` 100%, `cmd/build-dictionary` 82.4% |
Upstream schema auto-detected (`words.word` / `lang_code`) with no flag override needed,
retiring the phase-1 "unknown upstream schema" risk. Upstream asset pinned by SHA-256.
**Four defects were found and fixed in alias generation, three of them only visible by
inspecting real output rather than fixture tests:** `quý`→`qúy`, `gì`→`gỳ`,
`hoàn`→`hòan` (tone shift must be confined to open syllables), and an i/y rule so broad it
invented ~340 non-words such as `chức vỵ`. A fifth, opposite defect was found by review:
varying one syllable at a time never produced `hóa lý`, the spelling most people type.
Variants are now generated as a cross product.
A Vietnamese phonotactic check (closed onset/nucleus/coda inventories) now rejects
loanwords the multilingual source tags as Vietnamese — `credit card`, `world cup`,
`come out`. Its first version wrongly rejected the entire `gì`/`gỉ`/`gìn` family by
greedily matching the `gi` digraph; it backtracks over onset candidates now.
## Phase 4 Outcome (2026-09-04)
The wire contract is fixed and generated into both languages. Detail in
[`phase-04`](./phase-04-protobuf-contract-and-codegen.md#phase-4-outcome-2026-09-04).
Enum values carry buf's mandated enum-name prefix (`REJECT_REASON_TOO_FEW_SYLLABLES`, not
`TOO_FEW_SYLLABLES`), so `buf lint` passes with no carve-out; `protobuf-es` strips the prefix,
leaving the JS side reading `RejectReason.WRONG_LINK`. Two fields were added that the plan
sketch lacked: `PlayedWord.typed`, because the engine keeps the player's raw input alongside
the canonical spelling and the UI has to show a correction happened, and
`REJECT_REASON_GAME_OVER`, because `game.ReasonGameOver` exists and the exhaustiveness test
correctly refused a wire contract without it.
`web/` now exists as a bare npm package — protobuf and Vitest only — because phase 4's own
JS codegen and cross-language test need somewhere to live. Phase 6 layers SvelteKit on top.
Both suites read the same 17 binary fixtures in `proto/testdata/`, emitted by the Go tests.
Review caught four guards that were reporting green while protecting nothing — a `buf
breaking` baseline that cannot resolve on a PR checkout, a sync check blind to untracked
files, and two tests whose oracles could drift with the code they checked. All four were
fixed and then negative-tested by deliberately breaking each one.
## Phase 5 Outcome (2026-09-05)
The transport layer is in: rooms, turn timers, the bot as a virtual player, reconnect, and
a single binary that serves both the API and the frontend. Detail in
[`phase-05`](./phase-05-go-websocket-server-and-rooms.md#phase-5-outcome-2026-09-05).
The concurrency contract holds structurally rather than by convention: the room goroutine
starts before anyone is seated and seating itself is a message, so reading `run()` is a
complete proof that one goroutine owns each engine. The bot searches a frozen board rather
than the live engine, and reads carry no deadline — liveness is ping-based, because a read
timeout cannot distinguish a healthy player idling in the lobby from a dead socket.
Seven defects were found and fixed: three by writing the tests, three more by review, and
a seventh while fixing those. The serious one was authorization — the hub bound a joiner
to a seat before the room decided whether to seat them, so anyone holding a room code
could resign or play on a seated player's behalf. Since the room code is the only
credential online 1v1 has, that was a phase-7 release blocker caught a phase early.
## Phase 6 Outcome (2026-09-05)
The vs-bot UI is built and served by the Go binary. Detail in
[`phase-06`](./phase-06-sveltekit-frontend.md#phase-6-outcome-2026-09-05).
The store is a reducer over `ServerMessage` and computes nothing: validity, turn order and the
result are read from the wire, which is what lets one screen serve both the bot and, in phase
7, online play. The word field is deliberately uncontrolled, because a Telex input method
composes a diacritic over several keystrokes and writing the value back cancels it. The
countdown is drawn against the server's clock and settles 300ms early, so the ring never claims
more time than the server allows.
Three defects were found by building and five more by review. The serious one was that the game
lifecycle was inferred from the game model rather than owned: the screen asked for a bot game
whenever the board was idle, and clearing the board for a rematch is exactly that condition, so
every rematch started two server-side rooms that then destroyed each other. A request is stored
intent now, with tests over the wiring whose absence let it through. Review also caught a word
silently dropped when typed during a reconnect, a protocol bump that would have put every open
tab into a reconnect loop, an opening turn counted against the device clock, and an
`index.html` cacheable across a deploy.
Four criteria need a real browser and stay open: one-handed mobile use, Telex diacritic entry,
the absence of a theme flash, and reconnect after the server is killed mid-game.
## Phase 7 Outcome (2026-09-05)
Online 1v1 is playable end to end and the game ships as one 24 MB container image. Detail in
[`phase-07`](./phase-07-online-1v1-and-release.md#phase-7-outcome-2026-09-05).
The rematch needed the one thing phase 5 deliberately did not do: let a room outlive its game.
It now does, but only while both seats hold live connections, so a bot room still closes the
moment its game ends. There is no decline message — leaving is the decline, which the server
already learns from the socket closing.
Building the browser suite found four defects the unit tests could not: a player who refreshed
mid-game could not rejoin, a restored opponent never cleared the other player's disconnect
banner, a socket that failed without closing was invisible to the client, and a nickname typed
on the online screen never reached the server because the handshake had already gone.
Two testing tools turned out not to work here and are documented where the next person will
look: offline emulation does not cut a loopback socket, and a routed socket's close does not
reliably reach the server, so the grace window never starts.
Review then reproduced three more. Two follow from the new state this phase introduced — a room
that outlives its game — and one predates it: a game ending on the turn clock never offered a
rematch although the button was there, a second connection presenting the same resume token
disconnected the player who had not left, and rooms leaked whenever one connection created
several. All three now have regression tests, and the two reachable through the room loop were
negative-tested by reverting the fix.
## Open Questions
1. Does the losing player see the words the bot *could* have played (a teaching feature), or just the result? Plan currently assumes just the result.
2. Should the builder cap maximum syllables (e.g. reject 5+ syllable entries as phrases rather than words)? Plan currently applies no upper bound; the cap exists in the builder as a flag if playtesting says otherwise.
3. Domain name / TLS certificate source for the VPS deployment.
4. Should the turn clock pause while a seat is inside its reconnect grace window? As
built the grace window (30s) is longer than the turn limit (20s), so a player who
drops on their own turn always loses on time before the window closes — the grace is
unreachable for the player most likely to need it. Their reconnect is now told
`game_already_over` rather than left waiting, but the underlying policy is unresolved.
5. Should bot rooms be registered in the joinable room namespace at all? They have no code
to share, and excluding them removes a whole class of stranger interference for free.
6. Is closing the connection the right response to a full outbox mid-game? 32 queued
frames is not much on a lossy mobile link, and the opponent currently sees that player
as having left.
plans/260907-1551-room-lobby-ready-system/plan.md
-70
View File
@@ -1,70 +0,0 @@
# Room lobby with owner/guest roles and ready-up
**Status:** delivered
## Outcome
An online room is a persistent lobby rather than a wrapper around one game. The
first occupant is the owner, the second is the guest. The guest readies; the
owner starts once they are; a finished game returns both to the lobby, where the
guest must ready again. Players can leave, the owner can kick an unready guest,
and an owner who leaves hands the room to whoever is left.
## Decisions taken
- **The owner has no ready flag.** Pressing Start *is* their readiness, so only
the guest can be ready, blocked from leaving, or kicked.
- **Room lifetime:** closes when the last seat empties, or after 10 minutes in
the lobby with no game started. A running game is bounded by the turn clock.
- **The rematch handshake is retired.** `RequestRematch`/`RematchState` stop
being sent and their proto tags are reserved; the lobby is the only place a
next game is negotiated.
- **A kick only frees the seat.** The kicked player may rejoin with the code.
## Constraints
- Proto changes are additive; retired tags go to `reserved`, never reused.
- The engine stays two-player; `seats [2]*seat` stays the room's shape.
- Readiness, roles, start and kick are server state. The client renders one
snapshot message and derives nothing.
- Only the room goroutine touches room state; new actions arrive as inputs.
- Every lobby action authorizes against the seat, not the claimed id: a room
code is a shared secret by design.
- Vietnamese copy stays in `web/src/lib/i18n/vi.js`; error codes stay UI keys.
## Non-goals
Bot games (start immediately, no readiness), spectators, more than two seats,
kick bans, accounts, and resuming into a room whose game already finished.
## Acceptance criteria
1. A guest who joins sees a lobby, not a board. No engine exists until Start.
2. Start is refused unless the guest is seated, connected and ready.
3. Game over returns both to the lobby with the guest unreadied; a second game
needs a fresh ready.
4. Leave while ready is refused; leave while unready frees the seat and the room
survives.
5. Kicking a ready guest is refused; kicking an unready one frees the seat and
tells them why.
6. An owner who leaves, or drops for good, promotes the other player, who is
unreadied and can start once someone joins.
7. The last player out closes the room; an idle lobby closes on the TTL.
8. A reconnect lands back in the lobby with roles and readiness intact.
9. Start racing an unready or a leave, and a kick racing a ready, resolve to an
error for the loser and never a half-started game.
## Phases
1. **Wire contract.** `SetReady`, `StartGame`, `KickPlayer`, `LeaveRoom`,
`RoomState`; reserve the rematch tags; regenerate both trees; extend the
cross-language fixtures.
2. **Room lifecycle.** Owner/guest roles, ready flag, start, kick, leave,
promotion, idle TTL, resume into a lobby, retire the rematch path. Route and
rate-limit the new messages in `session.go`.
3. **Client.** A `lobby` phase in the game store driven by `RoomState`, a Lobby
component, retire `RematchPrompt`, copy, and the `/online` route wiring.
4. **Tests.** Room tests for each acceptance criterion, store tests for the new
phase, e2e for join → ready → start → play → back to lobby, leave, kick and
promotion.
5. **Docs.** The online-play section of the README.
plans/260907-1636-room-chat/phase-01-start.md
-118
View File
@@ -1,118 +0,0 @@
---
phase: 1
title: "Wire contract"
status: completed
priority: P1
effort: "1h"
dependencies: []
---
# Phase 1: Wire contract
## Overview
Adds the three messages chat needs to `proto/noitu/v1/game.proto`, regenerates
both trees, and extends the cross-language fixtures so the Go and JavaScript
clients are checked against one artifact.
## Requirements
- Functional: a client can send text; the server can deliver one message to
each recipient rendered from their side, and replay a bounded history.
- Non-functional: additive only. The current maxima are `ClientMessage` 12
(`leave_room`, with 8 reserved) and `ServerMessage` 12 (`room_state`, with
2, 3 and 11 reserved), so the next free tags are 13 on the client oneof and
13 and 14 on the server oneof.
## Architecture
```proto
// SendChat is one line of text from a seated player to the other.
message SendChat {
string text = 1;
}
// ChatMessage is one line as one recipient sees it. Rendered per recipient
// like every other room message: from_me is the only field that differs, and
// it is what lets the client style its own words without matching names.
message ChatMessage {
bool from_me = 1;
// Server-sanitized, as everywhere a name is shown. Empty when the author
// has left the room: the seat they spoke from may be somebody else's now,
// and a name outlives neither. The client labels an empty author itself.
string author = 2;
string text = 3;
// Server clock. The client renders it; it never orders by its own clock.
int64 sent_unix_ms = 4;
}
// ChatHistory is the whole panel, oldest first, sent when a player is seated
// in a room or resumes into one. A snapshot rather than a replayed stream, for
// the same reason RoomState is one: a client that missed frames is correct
// again from the next one instead of having to catch up on events.
//
// Scoped to the recipient: it carries only what was said while they held their
// seat, which is why it is built per seat rather than broadcast.
message ChatHistory {
repeated ChatMessage messages = 1;
}
```
Wired into the oneofs as `send_chat = 13` (client) and `chat_message = 13`,
`chat_history = 14` (server).
`ProtocolVersion` does **not** change, and the honest reason is not that an
old client is unaffected — it is pushed `ChatMessage` frames it never asked
for, because delivery is unconditional and there is no capability negotiation.
It drops them silently: the generated union yields `case: undefined` and the
store's `apply` switch has no `default` arm. The degradation is one-way chat
for the length of a deploy, and it is accepted rather than unnoticed. Phase 3
adds the `default` arm so the next contract addition is at least visible in a
console. If one-way chat is ever *not* acceptable, bumping `ProtocolVersion` is
the only mechanism the codebase has, and it forces a stale tab to reload.
## Related Code Files
- Modify: `proto/noitu/v1/game.proto`
- Modify (generated, do not hand-edit): `server/gen/noitu/v1/game.pb.go`,
`web/src/lib/proto/noitu/v1/game_pb.js`, `web/src/lib/proto/noitu/v1/game_pb.d.ts`
- Modify: `server/internal/wsapi/wire_test.go` — add `client_send_chat`,
`server_chat_message`, `server_chat_history` variants
- Create (generated): `proto/testdata/client_send_chat.bin`,
`proto/testdata/server_chat_message.bin`, `proto/testdata/server_chat_history.bin`
- Modify: `web/tests/game-wire.test.js`
## Implementation Steps
1. Add the three messages and their oneof arms to `game.proto`.
2. Lint and regenerate: `go run github.com/bufbuild/buf/cmd/buf@v1.58.0 lint`
then `... generate` from the repo root (no global `buf` needed).
3. Add the three variants to `wire_test.go`. Give them Vietnamese text with
diacritics and a real millisecond timestamp — the JavaScript runtime
surfaces `int64` as a bigint, and a zero value would skip that. Give
`ChatHistory` two messages: one from each side, and one with an empty
`author`, so the repeated field, both `from_me` values and the departed-author
case are all exercised.
4. Regenerate fixtures: `cd server && go test ./internal/wsapi -update`.
5. Assert the new fixture in `web/tests/game-wire.test.js`, including that
`sentUnixMs` arrives as a bigint and that an empty `author` survives the
round trip.
## Success Criteria
- [x] `buf lint` passes and both generated trees are current.
- [x] `TestVariantTablesCoverEveryOneofArm` passes, so every new arm is covered.
- [x] `go test ./internal/wsapi` and `npm test` both decode the new fixtures.
- [x] `git diff` on the generated trees shows only the new messages.
## Risk Assessment
Low. The one thing to get wrong is a tag collision with a reserved number;
`buf lint` catches it. If a later phase finds `ChatHistory` unnecessary, the
tag is reserved rather than reused — the cost of carrying it is one unused
message, not a migration.
The `author`-empty convention is load-bearing for Phase 2: if Phase 2 stores
and sends a departed player's name instead, two different people can appear
under one attribution. Signal: the Phase 4 vacate tests. Response: the
convention is stated here and tested there; keep them in step.
plans/260907-1636-room-chat/phase-02-room-chat-server.md
-241
View File
@@ -1,241 +0,0 @@
---
phase: 2
title: "Room chat"
status: completed
priority: P1
effort: "4h"
dependencies: [1]
---
# Phase 2: Room chat
## Overview
The server side: authorize the sender's seat, sanitize and cap the text,
rate-limit the sender without endangering the recipient, keep the last 20
messages per room, deliver each one rendered per recipient, and replay to a
seated player what was said while they held that seat.
## Requirements
- Functional: a seated player in a non-bot room can send text to the other
seat; both see it. A seating and a resume replay the recipient's own slice of
the history. History outlives a game and dies with the room.
- Non-functional: bounded memory per room; text sanitized before it is stored;
chat rate-limited on its own budget so talking never costs a move, and sized
so that talking cannot close the other player's session either; chat is not
room activity, so the idle timer still bounds the room.
## Architecture
**Sanitizer.** `sanitizeNickname` is close to the filter chat needs — NFC,
control and format characters dropped, whitespace collapsed, rune cap — but two
things separate them. Its cap and its empty-input fallback are nickname policy,
and its character filter admits combining marks: `unicode.IsPrint` counts marks
as printable and NFC does not discard uncomposable ones. At 20 runes that is
cosmetic; at 200 it is a glyph cluster hundreds of lines tall that escapes the
panel and covers the board, and it would sit in the history being replayed.
Extract the shared body into `sanitizeText(raw string, maxRunes, maxMarks int)
string` in `nickname.go`, returning `""` when nothing survives:
- NFC, then the existing control/format/non-print filter;
- **new:** drop `unicode.Mn`/`unicode.Me` runes beyond `maxMarks` consecutive
after a base rune (2 is enough for any Vietnamese cluster);
- collapse whitespace with `strings.Fields`, then the rune cap.
`sanitizeNickname` becomes that call plus the `defaultNickname` fallback and the
`denylist` hook. Its existing tests must pass unchanged — Vietnamese names carry
at most two marks per base rune, so the mark budget is invisible to them.
Collapsing whitespace also means a chat line cannot contain a newline, so no
message can be made to occupy a column of its own in the panel.
**Constants** (`room.go`, beside the other room constants):
```go
// chatHistoryLimit is how many messages a room keeps, and the same window the
// client holds. Enough to catch up on after a reload, few enough that a room
// that lives all day cannot grow.
const chatHistoryLimit = 20
// maxChatRunes caps one message. Runes, not bytes: Vietnamese is multi-byte
// and a byte cap would cut it far shorter than a Latin one.
const maxChatRunes = 200
// maxChatMarks is how many combining marks may follow one base rune. Two
// covers every Vietnamese cluster; more is a stacking attack, not a word.
const maxChatMarks = 2
```
**Rate limit** (`session.go`): a third bucket beside `submitLimiter` and
`roomLimiter` — `chatLimiter = newBucket(chatsPerSecond, chatBurst, ...)` with
`chatsPerSecond = 1`, `chatBurst = 3`. Its own budget so talking never costs a
move, and sized against the recipient rather than the sender: `outboxCap` is 32
and a single `Write` can stall for `writeTimeout` (10s), so the existing 5/s of
lobby and move traffic plus 1/s of chat stays comfortably inside the window a
stalled reader can drain.
**Delivery is best-effort.** `session.send` closes a session whose outbox is
full — correct for game frames, wrong for chat, because it hands one player a
way to disconnect the other into a grace-window abandonment loss. Add
`session.trySend(m)` that takes the same encode path but, on a full outbox,
logs and returns `false` instead of calling `close()`. Chat frames — and only
chat frames — go through it. A dropped chat frame is recoverable: the next
resume replays the history.
**Input.** `chatInput{sess *session, player game.PlayerID, text string}` on the
room's existing channel. In `dispatch`, gate on `chatLimiter` and `currentRoom()`
exactly as `toRoom` does, and answer a refused enqueue: `if !r.send(chatInput{…})
{ s.send(errorMsg("busy")) }`, matching `handleSubmit`. Nothing about the room's
own state is read on the session goroutine — `r.strategy` in particular is
room-goroutine-only, so the bot-room refusal belongs in the handler, not here.
**`handleChat` opens with authorization, like every other room mutation:**
```go
func (r *room) handleChat(m chatInput) {
// The seat, not the claimed id. A connection the room has already
// retired — kicked, or replaced by a reconnect — can still have a frame
// in flight, and its seat may belong to somebody else by the time the
// room drains it.
if !r.occupies(m.sess, m.player) {
m.sess.send(errorMsg("not_your_seat"))
return
}
if r.strategy != nil {
m.sess.send(errorMsg("not_in_a_room"))
return
}
…
}
```
**Storage.** On the room:
```go
// chat is the room's recent conversation, oldest first, capped at
// chatHistoryLimit. It outlives each game the way the room does.
chat []chatEntry
// chatSeq counts every message the room has accepted, ever. A seat records
// the value it was filled at, which is what scopes a replay to what that
// player was actually present for.
chatSeq uint64
type chatEntry struct {
seq uint64
// author and name are cleared together when the seat is vacated: the
// words stay, the attribution does not. Keeping the name would let the
// next person to request that nickname inherit a stranger's messages,
// since distinguish() only compares against the seat that is occupied.
author game.PlayerID
name string
text string
at time.Time
}
```
On `seat`: `chatFrom uint64`, set to `r.chatSeq` when the seat is filled
(`handleCreate`, `handleJoin`, `handleStartBot` for symmetry). A resume does not
touch it — the seat persists, so the player gets their conversation back.
`vacate` clears both `author` and `name` on that seat's entries. `beginGame`
does **not** touch the history: the room's conversation is not the game's.
**Idle timer.** `resetIdleTimer()` runs at the tail of every run-loop iteration,
so a chat message would restart the ten-minute lobby window and a room could be
held open forever by typing into it once every nine minutes. Chat is explicitly
not activity: track it per iteration —
```go
case chatInput:
r.handleChat(m)
// Deliberately no resetTurnTimer and no idle reset: talking is not
// playing, and the room's ten-minute bound has to survive a
// conversation.
idleActivity = false
```
— and guard the tail call with it. Skipping the reset leaves the running timer
alone, which is what "the clock keeps ticking" means.
**Replay.** `chatHistoryFor(s *seat)` sends one `ChatHistory` holding the
entries with `seq >= s.chatFrom`, oldest first. Called from `handleCreate`,
`handleJoin`, and `handleResume` — **before** `handleResume`'s
`if r.inLobby() { return }`, or a lobby refresh (the commonest resume) never
gets one. `handleCreate` sends an empty history on purpose: it is what
overwrites a stale array in a client that has been in another room.
The history therefore *precedes* that input's `RoomState`, which is broadcast
from the run-loop tail after the handler returns. That is fine and the client
must not depend on the order: `chatHistory` replaces wholesale and reads no
`RoomState`-owned field. Do not claim the opposite anywhere.
## Related Code Files
- Modify: `server/internal/wsapi/nickname.go` — extract `sanitizeText` with the
mark budget
- Modify: `server/internal/wsapi/room.go` — constants, `chatEntry`, `chat`,
`chatSeq`, `seat.chatFrom`, `handleChat`, `chatHistoryFor`, `vacate`
attribution clearing, the `chatInput` arm and the idle-activity guard in `run`
- Modify: `server/internal/wsapi/session.go` — `chatLimiter`, `trySend`, the
`ClientMessage_SendChat` arm in `dispatch`
- Modify: `server/internal/wsapi/codec.go` — builders if the room's existing
inline style does not fit
## Implementation Steps
1. Extract `sanitizeText(raw, maxRunes, maxMarks)` with the mark budget and
re-express `sanitizeNickname` on top of it. Run the existing nickname tests —
they must not change.
2. Add the constants, `chatEntry`, `chat`, `chatSeq` and `seat.chatFrom`; set
`chatFrom` at every seating site.
3. Add `chatLimiter` and `trySend`; add the `dispatch` arm with the `busy`
answer.
4. Implement `handleChat`: `occupies`, bot-room refusal, sanitize, drop empty,
append with the cap and a rising `chatSeq`, broadcast per recipient via
`trySend`.
5. Add the `chatInput` arm to `run` and the idle-activity guard around
`resetIdleTimer()`.
6. Clear `author` and `name` in `vacate`; leave `beginGame` alone.
7. Add `chatHistoryFor` and call it from `handleCreate`, `handleJoin`, and
`handleResume` before the lobby early return.
## Success Criteria
- [x] Two seated players exchange messages in the lobby and mid-game.
- [x] `from_me` is true only for the sender's own copy.
- [x] The 21st message pushes the 1st out.
- [x] A joiner's replay holds nothing said before they were seated; a resuming
player's holds everything they were present for.
- [x] A creator receives an empty history, so a stale client array is replaced.
- [x] A lobby resume replays, not only a mid-game one.
- [x] Text over the cap is truncated at a rune boundary; a 199-mark stack is
truncated; control and format characters never reach the other player.
- [x] A burst is refused with `too_fast` and leaves the move budget intact.
- [x] Sustained chat against a stalled reader drops frames rather than closing
that session.
- [x] A dropped `chatInput` answers `busy`.
- [x] An unseated connection, a connection the room has replaced, and a bot
room are all refused.
- [x] A room with chat traffic and no game still closes on the idle timer.
- [x] After a seat is vacated, that seat's messages carry no author and no
name.
## Risk Assessment
The authorization and the attribution rules are the two that bite. Skip
`occupies` and a retired connection speaks as a seat somebody else now holds;
keep the departed player's name and a newcomer can request that nickname and
inherit their messages, because `distinguish` only compares against the seated
opponent. Signal: the Phase 4 replaced-connection and vacate tests. Response:
both are a few lines at a single chokepoint — `handleChat`'s first statement
and `vacate`'s loop — so fix in place.
The mark budget is a judgement call: two marks per base rune covers Vietnamese,
but it is a rule about text, not a standard. Signal: a legitimate message losing
a mark. Response: raise `maxChatMarks`; do not remove it.
Memory is bounded by construction (20 entries × 200 runes per room). Room
*lifetime* is bounded by the idle timer only because chat is excluded from it —
that exclusion is the load-bearing part, not the byte count.
plans/260907-1636-room-chat/phase-03-client-chat-panel.md
-177
View File
@@ -1,177 +0,0 @@
---
phase: 3
title: "Client chat panel"
status: completed
priority: P1
effort: "4h"
dependencies: [1, 2]
---
# Phase 3: Client chat panel
## Overview
The panel the players actually use: a message list and an input, driven by the
store, rendered in the lobby and on the board. Expanded in the lobby, collapsed
behind an unread count during a game so it does not crowd a phone screen.
Depends on Phase 2 as well as Phase 1: every behavioural criterion below needs
a server that accepts `SendChat` and emits `ChatMessage`/`ChatHistory`.
## Requirements
- Functional: send a message; see both sides' messages in order with the
author's name, or a "left the room" label where the author is empty; a
replayed history replaces the list wholesale; an unread count while
collapsed; every server refusal visible, including in the lobby.
- Non-functional: text rendered as text, never markup; Enter must not send
mid-composition and the field's value must never be written back; the list
must not steal the page's scroll; the list must not grow without bound.
## Architecture
**Store** (`game.svelte.js`). One new field and two new arms:
```js
/**
* The room's conversation, oldest first, capped at the same window the server
* keeps so the two can never disagree about it.
*
* @type {{ fromMe: boolean, author: string, text: string, atMs: number }[]}
*/
chat: [],
```
- `chatMessage` → push one entry (`atMs: Number(value.sentUnixMs)` — the field
is an `int64` and arrives as a bigint, exactly like `deadlineUnixMs`), then
trim the front past `CHAT_WINDOW` (20, mirroring `chatHistoryLimit`).
- `chatHistory` → replace the array wholesale. A snapshot replaces; it never
merges. It also resets the unread count: the player is being resynchronised,
not shown new mail.
- Add a `default:` arm to the `switch` that logs an unhandled payload case, so
the next protocol addition degrades visibly instead of silently (Phase 1).
**Clearing is the subtle part.** `chat` joins the `kept` set, because a game
starting must not wipe the conversation — but `kept` means "survives
`reset()`", and the online route calls `reset()` on *mount and unmount* as well
(`online/+page.svelte`), so `kept` alone would carry room A's conversation into
room B. Two independent fixes, both cheap:
1. A `clearChat()` on the store, called in the online route's mount and
unmount effect alongside `reset()` — leaving the screen is leaving the room.
2. Phase 2 sends a `ChatHistory` on every seating including `handleCreate`, so
a stale array is overwritten by the server's truth even if the route forgets.
`leave()` clears it too, as it clears everything.
**`ChatPanel.svelte`** — props `{ collapsible?: boolean, onsend: (text: string) => void }`.
- Message list: `<ol>` with `overflow-y: auto` in its own scroll container,
`class:mine={entry.fromMe}`, the author's name shown only on the other side's
messages, `t.chatAuthorLeft` where `author` is empty, and time as `HH:mm` from
`atMs`.
- Scroll: oldest-first, so it scrolls to the **bottom** on a new message, and
only when already near it so it cannot yank a player out of scrollback. Write
this explicitly — `ChainHistory.svelte` is *not* the precedent: it renders
newest-first and scrolls unconditionally to `top: 0`.
- Input: **one-way mirroring only.** `oninput={(e) => (draft = e.currentTarget.value)}`
and never write `draft` back to the element — `WordInput.svelte` is
deliberately uncontrolled because writing the value back cancels Telex
composition and mangles the accent, and a `bind:value` here would reintroduce
exactly that. Keep its `oncompositionstart`/`oncompositionend` submit guard.
- Send is enabled when the draft is non-blank after trimming **and** stripping
zero-width and format characters — the same emptiness the server tests for
after sanitizing, which is what makes the server's silent drop unreachable
rather than merely unlikely.
- Length: check the rune count (`[...draft].length`) in the send handler, not
`maxlength` — `maxlength` counts UTF-16 code units, not runes, and interferes
with composition in some engines. The server truncates anyway; this is only
so the player is not surprised.
- Interpolated with `{entry.text}`, never `{@html}`.
- `collapsible`: a header button toggling the list, showing an unread count.
Unread is client-only state — messages that arrived while collapsed, reset on
expand and on a history replace. Per viewer, so nothing about it goes on the
wire.
**Error surface.** `Lobby.svelte` renders no `game.state.error` today, and the
online route's error paragraph exists only in the pre-room branch — so a
`too_fast`, `busy` or `not_your_seat` refusal in the lobby is invisible. Render
the store's error inside `ChatPanel` (dismissible, `role="alert"`), which covers
every phase the panel appears in and incidentally gives the lobby's own
refusals — `must_unready_first`, `player_is_ready`, `not_the_owner` — somewhere
to land.
**Placement** (`online/+page.svelte`):
- `phase === 'lobby'` → `<ChatPanel onsend={say} />` under `<Lobby />`.
- `phase === 'playing' | 'over'` → `<ChatPanel collapsible onsend={say} />`
passed to `GameBoard` as a `chat` snippet, rendered below the chain.
`GameBoard` already takes optional snippets (`banner`, `gameOver`), so the
bot route passes none and gets no panel — which is how the non-goal stays a
non-goal rather than a runtime check.
**Copy** (`vi.js`): `chatTitle: 'Trò chuyện'`, `chatPlaceholder: 'Nhắn tin…'`,
`chatEmpty: 'Chưa có tin nhắn nào.'`, `chatUnread: '{n} tin mới'`,
`chatAuthorLeft: 'Đã rời phòng'`. Reuse `t.submit` for the send button.
## Related Code Files
- Create: `web/src/lib/components/ChatPanel.svelte`
- Modify: `web/src/lib/stores/game.svelte.js` — `chat`, two arms, the `default`
arm, `kept`, `clearChat()`, `CHAT_WINDOW`
- Modify: `web/src/lib/ws/messages.js` — `sendChat(text)` builder
- Modify: `web/src/lib/components/GameBoard.svelte` — a `chat` snippet prop
- Modify: `web/src/routes/online/+page.svelte` — `say(text)`, `clearChat()` in
the mount/unmount effect, the panel in both phases
- Modify: `web/src/lib/i18n/vi.js`
## Implementation Steps
1. Add the `sendChat` builder.
2. Add `chat`, the two arms with the window trim, the `default` arm,
`clearChat()` and the `kept` entry to the store. Write the store test for
`atMs` being a number here, not in Phase 4 — `initialState()` is typed
`@returns {any}`, so `svelte-check` cannot catch a bigint or a misnamed
field, and the `deadlineUnixMs` precedent is guarded by a test for exactly
that reason.
3. Write `ChatPanel.svelte`: list, then the input with one-way mirroring and the
composition guard, then the collapsible header, then the error line.
4. Add the `chat` snippet to `GameBoard`, render the panel from the online route
in both phases, and call `clearChat()` where the route calls `reset()`.
5. Add the copy.
6. `npm run check`, then `npm test`.
## Success Criteria
- [x] A message typed in the lobby appears on both screens, in order.
- [x] Own messages are visually distinct; the other player's carry their name;
a departed author's carry the "left the room" label.
- [x] A reload shows that player's conversation, and the client list matches the
server's window exactly — no duplicate, no drift.
- [x] The list stops growing at the window; the unread badge cannot exceed it
and is reset by a history replace.
- [x] Navigating away and creating a new room shows an empty panel.
- [ ] Enter mid-composition does not send, and a Vietnamese word typed through
Telex arrives whole. *(guard implemented; harness cannot synthesise
composition events — see plan.md, Verification gaps)*
- [x] Send stays disabled for whitespace and for a pasted zero-width character.
- [x] A message containing `<b>x</b>` renders as those characters.
- [x] A `too_fast` refusal is visible in the lobby.
- [x] The panel is collapsed on the board and shows an unread count; the bot
screen has no panel at all.
- [ ] No horizontal page scroll at 360px wide; the list scrolls, the page does
not. *(bounded by CSS, checked by hand rather than asserted)*
- [x] `npm run check` reports zero errors.
## Risk Assessment
The composition guard is the one that bites: without it Vietnamese input is
unusable and the bug reads as "chat drops words". The trap is reaching for
`bind:value` to drive the disabled state — that is the write-back the comment in
`WordInput.svelte` warns about. Signal: accents mangle while typing, or the
Phase 4 diacritic case fails. Response: one-way mirroring as specified; if the
disabled state proves fiddly, enable Send unconditionally and let the server
drop the empty message rather than reintroducing a controlled input.
The unread count is client-only, so if it proves fiddly it can be dropped
without touching the server or the contract.
plans/260907-1636-room-chat/phase-04-tests.md
-146
View File
@@ -1,146 +0,0 @@
---
phase: 4
title: "Tests"
status: completed
priority: P1
effort: "4h"
dependencies: [2, 3]
---
# Phase 4: Tests
## Overview
One test per success criterion in the plan, at the cheapest layer that can hold
it: Go for the rules, Vitest for the store, Playwright for the two things only
a browser proves — Vietnamese input and the panel not eating the board.
## Requirements
- Functional: every criterion in `plan.md` has a test that fails without the
feature.
- Non-functional: no sleeps; wait on state the server produced, as the existing
suites do.
## Architecture
**Harness first.** `payloadCase` (`wsapi_test.go`) needs `chat_message` and
`chat_history` arms before anything else — its own comment says a missing arm
makes every `await` time out with nothing to say about why. Add a `say(text)`
helper on `testClient`. Note `silentFor` lives in `regression_test.go`, not
`wsapi_test.go` — same package, so it is usable either way.
`await` **discards** every frame before the one it wants, so it cannot assert
ordering. Any ordering claim must be checked with `recv()` in sequence — and
per Phase 2 the only true ordering claim is that the history precedes that
input's `RoomState`, which the client is explicitly not allowed to depend on.
Do not write an ordering assertion at all; assert arrival, not sequence.
Chat is broadcast to both seats, so drain both sides before asserting on a
later frame — the discipline the lobby tests already needed.
**Go — `server/internal/wsapi/`.** Reuse `pvpLobby` and `pvpRoom`.
- `TestChatReachesBothSeatsRenderedPerRecipient` — `from_me` true for the
sender, false with the author's name for the other.
- `TestChatWorksInTheLobbyAndInAGame` — one message before Start, one after.
- `TestAJoinerSeesNothingSaidBeforeTheySatDown` — owner talks alone, guest
joins, guest's history is empty. The disclosure boundary.
- `TestChatHistoryIsReplayedOnResumeInTheLobby` — talk in the lobby, drop the
socket, resume, assert the conversation comes back. The commonest refresh,
and the case a hook after `handleResume`'s `inLobby()` early return misses.
- `TestChatHistoryIsReplayedOnResumeMidGame` — the same during a game.
- `TestCreatingARoomReplaysAnEmptyHistory` — what overwrites a stale client
array.
- `TestChatHistoryIsCappedAndOrdered` — 25 messages, the window holds the last
20, oldest first.
- `TestChatHistorySurvivesAGame` — talk in the lobby, play and finish a game,
the history is still there.
- `TestChatTextIsSanitizedAndCapped` — a zero-width joiner and a bidi override
are gone; a 300-rune Vietnamese message comes back 200 runes, still valid
UTF-8; **a 199-mark stack after one base rune comes back with at most two.**
- `TestEmptyChatIsDroppedWithoutAnError` — send whitespace and a lone
zero-width character; assert silence with `silentFor`, and that a real
message still works afterwards.
- `TestChatIsRateLimitedOnItsOwnBudget` — a burst earns `too_fast`, and a
`SubmitWord` immediately after is still accepted.
- `TestSustainedChatDoesNotCloseTheOpponentsSession` — the recipient's outbox
under pressure drops chat frames rather than being closed. Drive it against a
client that stops reading; assert the session survives and the game does not
end in abandonment.
- `TestChatFromAReplacedConnectionIsRefused` — resume a seat, then send on the
retired socket: `not_your_seat`. The `occupies` guard.
- `TestChatFromAKickedConnectionIsRefused` — kick a guest whose message is
already queued, seat a replacement, assert the message is refused rather than
attributed to the newcomer.
- `TestChatNeedsASeat` — a greeted connection in no room: `not_in_a_room`.
- `TestBotRoomHasNoChat` — refused, like the lobby actions.
- `TestChatDoesNotKeepARoomAlive` — short `IdleFor`; chat throughout; the room
still closes with `room_idle_closed`.
- `TestVacatedSeatKeepsItsWordsButLosesItsAuthor` — guest talks, leaves, a third
player joins: the old message arrives with `from_me` false, an empty author,
and **no name**, checked from the remaining player's side as well as the
newcomer's.
**Vitest — `web/tests/game-store.test.js`.**
- `chatMessage` appends in arrival order with `atMs` as a number, not a bigint.
- The list stops at the window: push 25, assert 20 and that the oldest is gone.
- `chatHistory` replaces rather than merges (apply twice, no duplicates) and
resets the unread count.
- `reset()` keeps the conversation, `clearChat()` and `leave()` drop it, and
two rooms in sequence do not carry messages over.
**Playwright — `web/e2e/pvp-game.spec.js`.**
- Two players talk in the lobby, both see it; they talk again mid-game.
- A Vietnamese word typed through the composition path arrives whole. Drive it
with `insertCompositionText` via CDP or an IME-style key sequence; if the
harness cannot fake composition, assert instead that a `compositionend`-then-
Enter sequence sends exactly once and note the gap.
- A message containing markup renders as text (`getByText('<b>x</b>')`).
- After a reload the guest still sees their conversation.
- Navigating home and creating a new room shows an empty panel.
- At 360×640 the board has no horizontal scroll with the panel present.
## Related Code Files
- Modify: `server/internal/wsapi/wsapi_test.go` — `payloadCase` arms, `say`
helper, the chat tests
- Modify: `web/tests/game-store.test.js`
- Modify: `web/e2e/pvp-game.spec.js`, `web/e2e/helpers.js` (a `chat` locator
group beside `board`)
## Implementation Steps
1. Add the `payloadCase` arms and the `say` helper first — without them every
chat `await` fails as a timeout with no diagnosis.
2. Add the Go tests; run `go test ./internal/wsapi -count=2` to catch a
frame-ordering flake early, and `-race` for the goroutine-boundary cases.
3. Add the store tests; `npm test`.
4. Add the e2e cases; run with `PLAYWRIGHT_CHANNEL=chrome` when Playwright's
own Chromium is not installed.
5. Full sweep: `go test ./... -race`, `npm test`, `npm run check`,
`npx playwright test`.
## Success Criteria
- [x] Every `plan.md` criterion maps to a named test.
- [x] `go test ./... -count=2 -race` is green.
- [x] `npm test` and `npm run check` are green.
- [x] The whole Playwright suite is green, including the pre-existing cases.
- [x] No test sleeps to wait for chat, and no test asserts frame ordering.
## Risk Assessment
Faking IME composition in Playwright may not be reachable with the current
harness. Signal: the composition case cannot be written without CDP. Response:
keep the weaker `compositionend`-then-Enter assertion, state the gap in the
final report, and rely on the unit-level guard — do not delete the case and
call it covered.
`TestSustainedChatDoesNotCloseTheOpponentsSession` is the awkward one to write:
it needs a client that stops reading without closing its socket. Signal: it
cannot be made deterministic. Response: assert the narrower property — that a
chat frame to a full outbox returns false from `trySend` — as a unit test on the
session, and say so rather than dropping the criterion.
plans/260907-1636-room-chat/phase-05-docs.md
-71
View File
@@ -1,71 +0,0 @@
---
phase: 5
title: "Docs"
status: completed
priority: P3
effort: "30m"
dependencies: [4]
---
# Phase 5: Docs
## Overview
Chat is user-visible behaviour and a change to the wire contract, so the two
documents that describe those get a paragraph each. Nothing else moves.
## Requirements
- Functional: a reader of the README learns that rooms have chat and what its
bounds are; the deployment doc's list of fixed timings stays true — which it
does only because Phase 2 excludes chat from the idle timer. If that decision
is ever reversed, this requirement is the thing that breaks first.
- Non-functional: no new documents, no duplication of what the code already
states — link to the constants rather than restating their values twice.
## Architecture
**`README.md`, "Online play".** After the lobby paragraphs, one short
paragraph: the two players can talk, the conversation belongs to the room
rather than to a game, a player is replayed what was said while they held their
seat (so a refresh restores it and a newcomer starts at silence), and text is
sanitized by the same filter that sanitizes nicknames before anyone sees it.
Say plainly that a bot game has no chat.
**`docs/deployment.md`.** The "one timing is not configurable" paragraph is
still true — say why it survived chat: the idle window measures *no game
started*, and chatting deliberately does not reset it, so a room full of
conversation still closes on schedule. Add the two fixed chat constants
(history window and rune cap) beside it. No new section, and no env var, since
none is being added.
Check while there: does anything in either document still describe the room as
one game's container? The lobby work already fixed the obvious cases, but chat
belonging to the room is a second claim resting on the same fact.
## Related Code Files
- Modify: `README.md`
- Modify: `docs/deployment.md`
## Implementation Steps
1. Write the README paragraph in the "Online play" section.
2. Add the constants note and the idle-window clarification to
`docs/deployment.md`.
3. Re-read both sections around the edits for a stale claim about rooms.
## Success Criteria
- [x] The README says chat exists, that it is per room, that a replay is scoped
to the recipient's own occupancy, and that bot games have none.
- [x] `docs/deployment.md` states that chat does not reset the idle window,
lists the fixed chat constants, and adds no env var that does not exist.
- [x] No other document contradicts either.
## Risk Assessment
Negligible on its own, but this phase is where a reversal of Phase 2's
idle-timer decision would surface as a documentation lie rather than a bug.
Signal: `TestChatDoesNotKeepARoomAlive` removed or skipped. Response: rewrite
the deployment paragraph in the same change, never after it.
plans/260907-1636-room-chat/plan.md
-210
View File
@@ -1,210 +0,0 @@
---
title: "Room chat"
description: "Text chat between the two players in an online room, in the lobby and during a game"
status: completed
priority: P2
effort: ""
tags: [online, wire-contract, frontend]
created: 2026-09-07
---
# Room chat
## Overview
The two people in an online room can talk to each other. Chat lives with the
room, not with a game: it works in the lobby while they agree on a game, during
the game, and after it. A player is replayed what was said while they were
sitting in their seat, so a refresh does not wipe their conversation.
The room already broadcasts per-recipient state (`RoomState`) and already
sanitizes untrusted display text (`sanitizeNickname`). Chat reuses both shapes
rather than inventing new ones.
## Decisions taken
- **Scope: the whole room.** One panel, available from the moment a player is
seated — lobby, game, and the lobby a finished game returns to.
- **History: the last 20 messages, scoped to the recipient's own occupancy.**
A seat records where the conversation stood when it was filled, and a replay
starts there. A player who reconnects gets their conversation back in full; a
player who joins a room starts at silence. The room still keeps only 20, so a
long-lived room cannot grow.
- **Abuse controls: server-side only.** A dedicated per-session token bucket, a
rune cap, and the existing sanitizer extended with a combining-mark budget.
No mute button, no filter list.
- **Chat is not room activity.** A chat message does not reset the lobby's idle
timer, so the ten-minute bound the deployment doc promises stays true and a
room cannot be held open indefinitely by typing into it.
- **A message that sanitizes to nothing is dropped without an error, and the
client makes that unreachable** by trimming and stripping zero-width
characters before it will enable Send. Every *other* refusal is answered and
shown.
- **A chat line is best-effort; a history is not.** A full outbox drops a
`ChatMessage` rather than closing the recipient's session — a lost line is
recoverable and closing a session costs the victim the game. `ChatHistory`
still goes through `send`, because it is the frame that corrects a whole
panel and there is nothing behind it.
- **A replayed history counts as what it holds, not as nothing.** A resync
keeps a folded panel's unread count honest instead of stranding it.
- **A vacated seat's messages keep their text and lose their author entirely** —
no id and no name — so neither the next occupant nor the player who stayed can
have a stranger's words attributed to them.
## Constraints
- Proto changes are additive; retired tags go to `reserved`, never reused.
- Chat text is untrusted input rendered in another player's browser. It passes
the sanitizer before it is stored or sent, and is rendered as text, never as
markup.
- Only the room goroutine touches room state; chat arrives as another input,
and every chat action authorizes with `occupies` on that goroutine — a room
code is a shared secret by design, and a connection the room has already
retired must not be able to speak as its old seat.
- Vietnamese copy stays in `web/src/lib/i18n/vi.js`; error codes stay UI keys.
- The chat input is typed with a Telex or VNI input method, so Enter must not
send mid-composition, and the field's value must never be written back into
the DOM node — see `WordInput.svelte`'s comment for why.
## Non-goals
Bot games (nobody to talk to), spectators, emotes or canned phrases, message
editing or deletion, typing indicators, read receipts, chat that outlives the
room, a mute or report control, and any profanity list.
## Goals
| # | Goal | Priority |
|---|------|----------|
| 1 | Two seated players can exchange text in the lobby and during a game | P1 |
| 2 | A reconnect replays what that player was present for | P1 |
| 3 | Text is capped, rate-limited and sanitized before anyone sees it | P1 |
| 4 | Chat cannot cost a player their session, their room, or their game | P1 |
| 5 | The panel does not crowd the board on a phone | P2 |
## Phases
| # | Phase | Status |
|---|-------|--------|
| 1 | [Phase 1: Wire contract](./phase-01-start.md) | Done |
| 2 | [Phase 2: Room chat](./phase-02-room-chat-server.md) | Done |
| 3 | [Phase 3: Client chat panel](./phase-03-client-chat-panel.md) | Done |
| 4 | [Phase 4: Tests](./phase-04-tests.md) | Done |
| 5 | [Phase 5: Docs](./phase-05-docs.md) | Done |
Phases run in order. 2 depends on 1; 3 depends on 1 and 2; 4 depends on 2 and 3.
## Success Criteria
- [x] A seated player can send a message and the other player sees it, in the
lobby and during a game.
- [x] Each side sees its own messages marked as its own, and the author's
server-sanitized name on the other's.
- [x] A player who reloads is shown their conversation again; a player who
joins a room is shown nothing said before they sat down.
- [x] History survives a game start and a game end; it dies with the room.
- [x] The client's list and the server's history hold the same 20 messages
after a replay, and the unread badge cannot outlive them.
- [x] A message longer than the cap is truncated at a rune boundary, and a
stack of combining marks is truncated with it.
- [x] Control and format characters cannot reach another player's screen, and
no message is rendered as markup.
- [ ] Every refusal is answered and visible: `too_fast` on a burst, `busy` on a
dropped input, `not_your_seat` from a retired connection — including in
the lobby, which today renders no error at all.
- [x] A burst does not consume the sender's move budget, and sustained chat
does not close the recipient's session.
- [x] Chat from a connection with no seat, from a connection the room has
replaced, and from a bot room are all refused.
- [x] A room with chat traffic and no game still closes on the idle timer.
- [x] A seat that is vacated leaves its messages with no author at all, so
neither player can be shown a stranger's words as somebody's.
- [ ] Enter mid-composition does not send a half-typed Vietnamese word.
## Verification gaps
Everything above is implemented. These paths are implemented but not proven by
a test, and are listed here rather than ticked:
- **Enter mid-composition** (criterion 13). The guard is `oncompositionstart` /
`oncompositionend` around a submit that reads the element, identical in shape
to `WordInput.svelte`'s. Playwright in this harness cannot synthesise
composition events, which the plan anticipated; covered by review only.
- **`busy` on a dropped input** (part of criterion 8). Filling a 32-slot room
inbox deterministically is not reachable from a test client. `too_fast` is
covered on both sides — refused on the server, visible in the lobby panel —
and `not_your_seat` is covered on the server.
- **`vacate`'s kick and grace-expiry paths.** The re-sync is asserted for a
voluntary leave only; the other two callers were traced by reading.
- **The departed-author label.** The server clears attribution and two tests
assert it; nothing asserts the client renders "Đã rời phòng" for it.
- **`ChatPanel.svelte` has no component test**, because the repo has no Svelte
component suite. Everything in that file is covered end-to-end or not at all.
- **The 360px board layout** (goal 5). Bounded by `max-height` and
`overflow-wrap: anywhere`, checked by hand rather than asserted.
## Risks
- **The panel eats the board on a phone.** Mitigation: collapsed by default
during a game with an unread count; the lobby shows it expanded. Signal: the
e2e board assertions start needing scrolling. Response: make it a sheet.
- **The unread badge is client-only state derived from a server counter.** Two
reviewers found the same defect in it by reading, twice: first counting a
capped list's length, then going quiet after a resync reset the counter under
it. It is now `chatCount` set to the replayed length and a `seenAt` that
follows a decrease. Signal: a badge that stops counting, or one that counts
what was already read. Response: the arithmetic is four lines in one
component — but add the store test before the fix, not after.
- **A client applies `ChatHistory` before it has a room.** The history is sent
from the handler and therefore *precedes* that input's `RoomState`, which is
broadcast from the run-loop tail. The client must not depend on the order —
`chatHistory` replaces wholesale and reads no `RoomState`-owned field.
Signal: a panel rendering with an empty room code. Response: render the panel
from `phase`, never from the presence of messages.
- **The combining-mark budget is a judgement call, not a standard.** Signal: a
legitimate Vietnamese message loses a mark it needed. Response: the budget is
one constant in one function — raise it; do not remove the cap.
## Red Team Review
### Session — 2026-09-07
**Findings:** 15 distinct, deduplicated from 28 raw across 3 reviewers
(15 accepted, 0 rejected)
**Severity breakdown:** 3 Critical, 7 High, 5 Medium
**Reviewers:** Security Adversary (Fact Checker), Failure Mode Analyst (Flow
Tracer), Assumption Destroyer (Scope Auditor). Verification tier: Full.
**Evidence filter:** every finding carried `file:line` citations; none rejected
for want of evidence.
| # | Finding | Severity | Disposition | Applied To |
|---|---------|----------|-------------|------------|
| 1 | `chat` in the store's `kept` set leaks one room's conversation into the next; `reset()` also runs on route mount/unmount, and `handleCreate` sends no history to heal it | Critical | Accept | Phase 2, 3, 4 |
| 2 | Replay ordering inverted — `broadcastRoomState` runs in the run-loop tail — and `handleResume` early-returns for a lobby resume, so the commonest refresh never replays | Critical | Accept | Phase 2, 4, plan.md |
| 3 | `handleChat` specified without `occupies`; the claim that `toRoom` does a seat check is false. Bot-room check cannot live in `dispatch` — `r.strategy` is room-goroutine-only | Critical | Accept | Phase 2, 4 |
| 4 | Chat resets the lobby idle timer, so a room can be held open forever and Phase 5's "timings stay true" is unsatisfiable | High | Accept | Phase 2, 4, 5 |
| 5 | A separate chat budget is additive to the existing ones, and a full outbox closes the recipient's session — sustained chat can disconnect an opponent into an abandonment loss | High | Accept | Phase 2, 4 |
| 6 | "Empty message is unreachable" is false — spaces or a pasted zero-width character pass the blank check | High | Accept (user decision: fix the client's blank check, keep the silent drop) | Phase 3, 4, plan.md |
| 7 | The 20-message replay discloses a private conversation to any stranger who redeems the code, and lets an owner pre-load a payload for joiners | High | Accept (user decision: replay scoped to the recipient's own occupancy) | Phase 2, 4, plan.md |
| 8 | The sanitizer drops control and format characters but not combining marks; at 200 runes that is a render bomb that persists in history | High | Accept | Phase 2, 4 |
| 9 | A dropped `chatInput` answers nothing, unlike every other input; and the lobby renders no error surface at all | High | Accept | Phase 2, 3, 4 |
| 10 | A controlled chat input cannot copy `WordInput`'s deliberately-uncontrolled composition guard verbatim; `maxlength` counts UTF-16 units, not runes | High | Accept | Phase 3 |
| 11 | The client array is unbounded (the cap is server history only) and the unread badge diverges from the list on a replace | Medium | Accept | Phase 3, 4 |
| 12 | Keeping a departed player's `name` lets a newcomer take that nickname — `distinguish` only checks the seated opponent — and contradicts Phase 1's own proto comment | Medium | Accept | Phase 1, 2, 4 |
| 13 | The `ProtocolVersion` justification is false: chat is pushed to old clients, which drop it silently for want of a `default:` arm | Medium | Accept | Phase 1, 3 |
| 14 | Phase 3's success criteria mostly require Phase 2's server, but its dependencies said `[1]` | Medium | Accept | Phase 3, plan.md |
| 15 | Four false citations: `ChainHistory` scrolls to top unconditionally; tag maxima are 12/12; `silentFor` lives in `regression_test.go`; `payloadCase` needs new arms — plus "JSDoc keeps the panel honest" when the store's state is typed `any` | Medium | Accept | Phase 1, 3, 4 |
### Whole-Plan Consistency Sweep
- Files reread: plan.md, phase-01-start.md, phase-02-room-chat-server.md,
phase-03-client-chat-panel.md, phase-04-tests.md, phase-05-docs.md
- Decision deltas checked: 7 — occupancy-scoped replay; chat excluded from idle
activity; best-effort chat delivery; `occupies` on every chat input; author
cleared entirely on vacate; client blank check strict; client list capped to
the server's window
- Reconciled stale references: 9 — the replay-ordering claim in plan.md Risks
and Phase 2; "last 20 on join" in plan.md Decisions, Goals and Phase 4; the
`toRoom` seat-check claim in Phase 2; the `ProtocolVersion` justification and
tag maxima in Phase 1; the `name`-retention rule in Phase 2 against Phase 1's
proto comment; the `ChainHistory` scroll precedent and the JSDoc claim in
Phase 3; Phase 3's dependency list; Phase 5's "timings stay true"
- Unresolved contradictions: 0
plans/260908-1006-multiplayer-rooms/phase-01-engine-elimination.md
-48
View File
@@ -1,48 +0,0 @@
# Phase 1 — Engine: elimination and standings
`server/internal/game/engine.go`, `state.go`.
The engine already carries `players []PlayerID` and advances `turnIndex` modulo
its length, so seating more than two costs nothing. What is two-player is the
*ending*: `opponentOf` picks the winner, and any single failure finishes the
game.
## What changes
- `alive []bool` parallel to `players`, plus `aliveCount`.
- `outOrder []PlayerID` — elimination order, first out first. Rank is derived
from it, so nothing has to be recomputed on read.
- `outReason map[PlayerID]EndReason` — why each player went out.
- `Turn()` returns the next **living** player; `advance()` skips the dead.
- `eliminate(p, reason)` marks a player out, appends to `outOrder`, and finishes
the game when one player is left.
- `expire(now)` eliminates the player to act, then keeps eliminating the next to
act while the position has no legal move — which is what stops a dead end from
costing every remaining player a full turn clock each.
- `NoMove()` and `Resign(p)` route through `eliminate` too. `Resign` works out
of turn: a player may give up while somebody else is thinking.
- `Alive(p) bool` and `Standings() []Standing` for the transport layer.
- Delete `opponentOf`.
```go
// Standing is one player's final placing. Rank 1 is the winner; a player
// eliminated later ranks above one eliminated earlier.
type Standing struct {
Player PlayerID
Score int
Rank int
Reason EndReason // EndNone for the winner
}
```
`State` gains `Standings []Standing` and `Alive map[PlayerID]bool`.
## Invariants worth a test
- Two players: every existing outcome is identical to today's.
- Four players: A times out → A out, turn passes to B on the **same syllable**,
scores and used set survive.
- A dead end closed by A eliminates B, C and D in one call and leaves A standing.
- `Resign` out of turn eliminates the resigner and does not disturb whose turn
it is, unless the resigner *was* the player to act.
- Rank ordering: survivor 1, then reverse `outOrder`.
plans/260908-1006-multiplayer-rooms/phase-02-wire-contract.md
-61
View File
@@ -1,61 +0,0 @@
# Phase 2 — Wire contract and codegen
`proto/noitu/v1/game.proto`, regenerated into `server/gen/` and
`web/src/lib/proto/`. `ProtocolVersion` → **2**.
## New messages
```proto
// PlayerSlot is one seat in the room, rendered for one recipient.
message PlayerSlot {
string player_id = 1; // "p1".."p4" — stable within a room
string name = 2;
bool is_me = 3;
bool is_owner = 4;
bool ready = 5;
bool connected = 6;
}
// PlayerScore is one player in a running or finished game.
message PlayerScore {
string player_id = 1;
string name = 2;
bool is_me = 3;
uint32 score = 4;
bool eliminated = 5;
bool connected = 6;
uint32 rank = 7; // final placing, 1 = winner; 0 while in play
}
// PlayerEliminated is one player leaving a game that is still running.
message PlayerEliminated {
string player_id = 1;
string name = 2;
bool is_me = 3;
GameEndReason reason = 4;
// What the position still had, sent only to the player who went out.
repeated string suggestions = 5;
}
```
## Changed messages
| Message | Change |
|---|---|
| `RoomState` | `reserved 2, 4, 5, 6, 7, 8` (`i_am_owner`, `i_am_ready`, `opponent_*`). Adds `repeated PlayerSlot players = 9`, `uint32 max_players = 10`, `uint32 min_players = 11`, `uint32 grace_ms = 12`. `room_code` and `can_start` stay. |
| `GameStarted` | Adds `repeated PlayerScore players = 7`, `string turn_player_id = 8`. |
| `TurnUpdate` | `reserved 6, 7` (`my_score`, `opponent_score`). Adds `repeated PlayerScore players = 9`, `string turn_player_id = 10`. `played` may now be absent — an elimination advances the turn without a word. |
| `GameOver` | `reserved 3` (`my_score`). Adds `repeated PlayerScore standings = 6`, ordered by rank. |
| `PlayedWord` | Adds `string player_id = 6`, so a four-way chain says who played what without matching names. |
| `KickPlayer` | Adds `string player_id = 1` — the owner now names a seat. |
| `OpponentLeft` | Retired. `ServerMessage` reserves tag 8; presence travels on `RoomState.players[].connected` with `RoomState.grace_ms`, which is broadcast mid-game already. |
| `ServerMessage` | `reserved 2, 3, 8, 11`; adds `PlayerEliminated player_eliminated = 15`. |
`i_am_owner` / `i_am_ready` go because the recipient's own row already carries
them: two encodings of one fact are two ways for a client to disagree with the
server.
## Regeneration
`buf` is not on this machine — `go install github.com/bufbuild/buf/cmd/buf@latest`
first, then `make proto`. Both generated trees are committed.
plans/260908-1006-multiplayer-rooms/phase-03-room-seats.md
-60
View File
@@ -1,60 +0,0 @@
# Phase 3 — Room: N seats, N grace windows
`server/internal/wsapi/room.go`, `codec.go`, `hub.go`, `session.go`,
`nickname.go`.
## Seats
- `maxPlayers = 4`, `minPlayers = 2`, `seatIDs = [maxPlayers]game.PlayerID{"p1".."p4"}`.
- `seats [maxPlayers]*seat`. `freeSeat`, `occupied`, `seatOf` already loop.
- `distinguish(name, taken)` becomes `distinguish(name, taken []string)` and
counts up (" 2", " 3", …) until the name is unused.
- Drop `otherNickname` / `opponentSeat`; add `seatsInOrder()` yielding the
occupied seats in seat order, which is also engine turn order.
## Lobby
- `canStart()` — no bot strategy, in lobby, at least `minPlayers` seated, owner
connected, every non-owner seat connected and ready.
- `lobbyStart` refuses with `need_more_players` / `player_offline` /
`not_everyone_ready`.
- `lobbyKick` reads `KickPlayer.player_id`: refuse `no_one_to_kick` when the
seat is empty, `cannot_kick_self` when it is the owner's own, and
`player_is_ready` when that player is ready — the existing rule, per target.
- `handleJoin` refuses a full room exactly as now.
## Grace windows
One timer is no longer enough: any number of seats can be inside their window at
once. Each `seat` gains `graceUntil time.Time`; `resetGraceTimer()` arms a single
timer for the earliest of them, and on fire every seat past its deadline is
expired together. That keeps one timer and one wakeup regardless of how many
players dropped.
A dropped seat mid-game is **not** skipped: the engine's turn clock runs for it,
so a player who drops on their own turn is eliminated by timeout before their
window is up. The window only decides whether they are still in the game after it.
## Eliminations
`applyEliminations(before int)` — compare the engine's `outOrder` length against
what it was, send each newly eliminated player's `PlayerEliminated` to every
seat (with `suggestions` only in the copy that goes to the player who went out),
then:
- game over → `broadcastGameOver` with standings;
- otherwise → `r.turnSeq++` and `broadcastTurn(nil)`, so every client learns the
new deadline and whose turn it is without a word having been played.
The room keeps `outWire map[game.PlayerID]noituv1.GameEndReason`, because
`OPPONENT_LEFT` is a transport fact the engine never learns: a grace expiry
eliminates with `EndResigned` and is reported as `OPPONENT_LEFT`.
`GameOver.reason` stays the game-level reason — the reason the **last**
elimination happened — which keeps 1v1 wording identical.
## Broadcasts
- `broadcastRoomState` builds `PlayerSlot` rows in seat order.
- `broadcastTurn(move *game.Move)` and `sendGameStarted` build `PlayerScore` rows.
- `gameOverFor` builds standings once and sets `is_me` per recipient.
plans/260908-1006-multiplayer-rooms/phase-04-client.md
-37
View File
@@ -1,37 +0,0 @@
# Phase 4 — Client: the room as a list of players
## Store — `web/src/lib/stores/game.svelte.js`
`opponentName` / `opponentScore` / `opponentPresent` / `opponentReady` /
`opponentConnected` / `isOwner` / `isReady` / `opponentLeft` all collapse into:
```js
/** @type {PlayerSlot[]} */ roomPlayers: [], // from RoomState
/** @type {PlayerScore[]} */ gamePlayers: [], // from GameStarted / TurnUpdate
/** @type {PlayerScore[]} */ standings: [], // from GameOver
maxPlayers: 4,
minPlayers: 2,
graceMs: 0,
turnPlayerId: '',
/** @type {{ playerId, name, isMe, reason, suggestions } | null} */
lastElimination: null,
```
with derived getters `me`, `isOwner`, `isReady`, `myScore`, `iAmEliminated`.
A `turnUpdate` with no `played` is an elimination advancing the turn — push
nothing to the chain, apply the rest.
## Components
| File | Change |
|---|---|
| `Lobby.svelte` | Renders `roomPlayers` plus empty slots up to `maxPlayers`. Kick button per row, on any guest who is not ready. Hints keyed off how many are seated and how many are ready. |
| `ScoreBoard.svelte` | A row per player, active row = `turnPlayerId`, eliminated rows struck through. Replaces the two-sided layout. |
| `PlayerStatus.svelte` | Replaces `OpponentStatus.svelte`: one line per disconnected player counting down `graceMs`, plus an "eliminated" banner for this player. |
| `GameOverPanel.svelte` | Standings table from `standings`; `myScore` read from the row where `isMe`. Suggestions still only for a player who did not win. |
| `ChainHistory.svelte` | Labels each entry with its player's name via `playerId`. |
| `GameBoard.svelte` | Drops `opponentLabel`; word input hidden while `iAmEliminated`. |
| `online/+page.svelte` | `kick(playerId)`. |
| `messages.js` | `kickPlayer(playerId)`, `PROTOCOL_VERSION = 2`. |
| `history-export.js` | Names each mover from the player list rather than "you / opponent". |
| `i18n/vi.js` | New keys: standings, eliminated, spectating, per-player disconnect, `need_more_players`, `cannot_kick_self`, `player_offline`. |
plans/260908-1006-multiplayer-rooms/phase-05-tests-docs.md
-30
View File
@@ -1,30 +0,0 @@
# Phase 5 — Tests and docs
## Go
- `game/engine_test.go` — the phase 1 invariants: four-player elimination order,
dead-end cascade, out-of-turn resign, standings ranks, and every existing
two-player case unchanged.
- `wsapi/wsapi_test.go` — a four-player room: join to full, fifth refused; start
refused until all ready; kick by id; a full game with two eliminations; an
eliminated player still receiving turn updates and chat.
- `wsapi/regression_test.go` — reconnect inside and outside the window with more
than two players seated.
- `wsapi/wire_test.go` — fixtures for the new and changed messages;
`go test ./internal/wsapi -update` rewrites `proto/testdata/`.
- `wsapi/convert_test.go` — unchanged mappings still exhaustive.
## JavaScript
- `tests/game-store.test.js` — the new projection, including a `turnUpdate` with
no `played`.
- `tests/game-wire.test.js` — decodes the regenerated fixtures.
- `tests/error-codes.test.js` — every server code has Vietnamese copy.
- `e2e/pvp-game.spec.js` — a three-player game through the browser.
- `e2e/reconnect.spec.js` — unchanged 1v1 path still passes.
## Docs
- `README.md`: "online 1v1 by room code" becomes rooms of two to four, and the
elimination rule joins the opening description.
- `docs/deployment.md` only if a knob changed — it did not.
plans/260908-1006-multiplayer-rooms/plan.md
-55
View File
@@ -1,55 +0,0 @@
# Rooms of two to four players
Status: **done** — 2026-09-08. Phases 1–5 delivered and verified, except the
Playwright suite, which could not run on this machine: the browser download
times out here. The specs are written and parse; they have not been executed.
A room seats up to `maxPlayers` (4, one constant) instead of exactly two. The
owner starts once every guest is ready; two seated players is enough. A player
who fails their turn is eliminated and the rest play on, last standing wins.
## The rules this settles
| Question | Answer |
|---|---|
| Room size | 2–4 seats. `maxPlayers` is one server constant, sent to the client in `RoomState` so the lobby renders whatever the server allows. |
| Starting | Owner only, and only when every other seated player is connected and ready. Two players is the minimum. |
| Kicking | Owner only, names a seat, refused when that player is ready — the existing rule, now per-target. |
| Failing a turn | The player who runs out of clock is **eliminated**; the syllable and the used set survive, the turn passes to the next living player. Last standing wins. |
| A dead-end position | The player to act still burns their clock and goes out, as today. Everyone after them is then eliminated at once rather than each waiting out a turn they cannot answer — which leaves the player who closed the position standing, exactly the 1v1 outcome. |
| Disconnecting mid-game | The seat's clock keeps running, so a player who drops on their own turn can time out and go out normally. Otherwise their reconnect window decides it: come back and play on, or be eliminated when it expires. |
| Being eliminated | They keep their seat and watch: live chain, live turns, chat still works, no word input. Everyone lands back in the same lobby when the game ends. |
| Game over | Full standings — every player, their score, ranked by finishing order (last standing first, then reverse elimination order). |
## Wire contract
This breaks the two-player shape of `RoomState`, `TurnUpdate`, `GameOver` and
`OpponentLeft`, so `ProtocolVersion` goes to **2** and the retired field tags go
to `reserved`. That is what the version number is for: an old client is refused
with a readable error rather than decoding a frame that means something else now.
## Phases
| # | Phase | Depends on |
|---|---|---|
| 1 | [Engine: elimination and standings](./phase-01-engine-elimination.md) | — |
| 2 | [Wire contract and codegen](./phase-02-wire-contract.md) | — |
| 3 | [Room: N seats, N grace windows](./phase-03-room-seats.md) | 1, 2 |
| 4 | [Client: the room as a list of players](./phase-04-client.md) | 2 |
| 5 | [Tests and docs](./phase-05-tests-docs.md) | 1–4 |
All five are complete. `go vet`, `go test ./... -race` and `npm test` pass;
`npm run check` reports no problems. `npx playwright test` is blocked on the
browser download, not on the code.
## Acceptance criteria
- A room seats 2–4; a fifth join is refused with `room_full`.
- Owner's start button is refused until every other seated player is ready.
- The owner can kick any named guest who is not ready.
- A 4-player game eliminates on timeout and continues; the last player standing
wins and everyone sees the same standings.
- An eliminated player still sees the board and can chat, and returns to the
lobby with the others.
- 1v1 behaviour is unchanged end to end, including reconnect and vs-bot.
- `go vet`, `go test -race`, `npm test` and the Playwright suite all pass.
plans/260908-1525-dictionary-corpus-switch/audit-proper-noun-drops.md
-184
View File
@@ -1,184 +0,0 @@
# Audit: proper-noun drops and the wiktionary-only corpus
Date: 2026-09-08. Phase 3 of `plan.md`. Every number here is reproducible from the
commands listed; the shipped `data/noitu.db` was copied aside before any build and is the
"old" side of every comparison.
## Commands
```sh
# from server/, after `make fetch-dict`
go run ./cmd/build-dictionary --merged ../data/undertheseanlp-words.jsonl \
--out /tmp/wik.db --report-drops /tmp/drops-wik.txt
go run ./cmd/build-dictionary --merged ../data/undertheseanlp-words.jsonl \
--sources hongocduc,wiktionary --out /tmp/hw.db --report-drops /tmp/drops-hw.txt
# sample: python, random.seed(1), random.sample(sorted(drops-wik), 100)
# casualties: (drops-wik − drops-hw) ∩ words(hw.db)
go test ./internal/bot/ -run RealCorpus -v # once per database at data/noitu.db
```
## Build result
```
rejected 97: contains punctuation
rejected 5074: fewer than 2 syllables
rejected 37: no Vietnamese letters
rejected 4747: only ever capitalized
accepted 22419 distinct words (sources: wiktionary)
```
22,419 vs the 22,310 the research report measured. The delta is the rule: the report
dropped any form containing an uppercase letter, the builder drops forms that *begin* with
one, so `tia X`, `bộ bài tây`, `chữ Hán`, `châu Á` survive. All are ordinary words.
## 1. Drop audit — 100 sampled, seed 1
Verdicts: **P** proper noun (correct drop) · **X** not a Vietnamese word (would be rejected
by the filter anyway) · **U** unclear · **C** common word (false positive).
| | | | | |
|---|---|---|---|---|
| a yun P | an hoà tây P | an minh bắc P | an thuận P | ba vinh P |
| ban cơ U | brao X | bàn đạt P | bành trạch P | bàu năng P |
| bá xuyên P | bình hiệp P | bình tấn P | bằng la P | chiềng sơ P |
| châu hội P | chù lá phù lá P | chư krêy P | cur X | côn đảo P |
| cơ kiều P | cẩm trung P | hùng vương P | hồng bàng P | irc X |
| khánh gia P | lão quân P | lương đài P | mặc dương P | nhơn hoà lập P |
| ninh lai P | ninh thạnh P | noong bua P | nội thôn U | phiếu hữu mai U |
| phù lảng P | phương cao kén ngựa U | phần lão U | quán hành P | quắc hương P |
| rã bản P | sam mứn P | sơn hạ P | sơn vy P | tam ngọc P |
| thanh liên P | thanh văn P | thiệu đô P | thái an P | thượng lâm P |
| thượng tiến P | thạch xuân P | thới quản P | tiên cát P | tiên thọ P |
| tri lễ P | triệu giang P | triệu việt vương P | trung tú P | trà khê P |
| trà kót P | trà linh P | trường khánh P | trường long P | trần đình thâm P |
| trọng do P | tuân lộ P | tân hưng P | tân mỹ P | tân nhựt P |
| tân trì P | tây hiếu P | tăng sâm P | tĩnh gia P | tịnh an P |
| tốt động P | vinh an P | việt–mường X | vân canh P | võ trường toản P |
| võ văn tồn P | văn môn P | văn quân P | văn đình dận P | vĩnh hoà hưng bắc P |
| vĩnh lương P | vĩnh lộc b P | vĩnh trinh P | vũ thư P | vạn thuỷ P |
| xuân mai P | xuân quan P | xá bung P | xá cẩu P | xá khắc P |
| xốp cộp P | yên cát P | ô qua U | đồng nai P | đổ rượu ra sông thết quân lính C |
**P 89 · X 4 · U 6 · C 1.** Gate is ≤2 common words in 100: **passes.** The bulk is
commune names, ethnonyms and historical persons — the content the rule exists to remove.
## 2. Case casualties — 215, listed in full
Words the wiktionary rows hold only capitalized, which another branch has lowercase. They
are dropped under the plan's wiktionary-only evidence rule. Full list:
`case-casualties.txt` in the session scratchpad; reproduced here because it is the
decision input.
an bình · an dân · an hảo · an khang · an lạc · ba tiêu · biển hồ · bu lu · **bàn là** ·
**báo đáp** · bát tiên · bình chuẩn · bình chân · bình khang · bình nghị · bình sa · bình
thanh · bình thuỷ · bình trị · bình tâm · bình văn · bình điền · bích đào · bóng chim tăm
cá · bông trang · bạch cung · bản nguyên · **bảo toàn** · bắc phong · bẻ quế · bố chính ·
**bồ đề** · bồng sơn · cao biền dậy non · cao kỳ · cao nhân · cao sơn · cao tổ · cao xanh ·
cao đường · chà và · chày sương · chánh hội · **chân mây** · châu lệ · châu thành · chí
thiện · chí thành · chính tâm · **chúa nhật** · chúa trời · chị hằng · con tạo · cát lũy ·
cát tân · cát đằng · **công bình** · công dã tràng · **công giáo** · **công nguyên** ·
**cơ đốc giáo** · cường thịnh · cầm đuốc chơi đêm · cầu lam · cầu lộc · cầu ô · cẩm châu ·
cẩm đường · cửu kinh · **cựu ước** · diêm vương · diêm vương tinh · dã hạc · dương quan ·
dương đài · **dường như** · **giao tử** · giấc mai · giọt châu · giọt tương · hoa cái ·
**hoa kiều** · **hoà đồng** · hoá công · hoả tinh · hán học · hán tộc · hán tự · hán văn ·
hạ thần · hải phòng · hải triều · hải vương tinh · hằng nga · **hết sảy** · **hệ mặt
trời** · hội gió mây · hội long vân · **hồi giáo** · **khổng giáo** · kim tinh · **kinh
thánh** · la ve · lâm viên · **lưỡi hái** · lạc hầu · mạnh thường quân · **mặt trời** · nam
lâu · **nguyên đán** · **nho học** · nhân kiệt · năm cha ba mẹ · nước dương · nếm mật nằm
gai · nợ như chúa chổm · phong thu · **phù phiếm** · **phật giáo** · phật học · phật pháp ·
phật tiền · phật tổ · phật tự · **phật đản** · quang phục · quyết tiến · quân thiều · **quả
đất** · **sao hôm** · sao hỏa · sao kim · **sao mai** · sao mộc · song mai · sơn cương ·
sơn lâm · sơn mai · sơn nguyên · sơn trung · sư tử hà đông · **sừng trâu** · tam dân · tam
phủ · tam sơn · thanh nghị · thanh phong · thanh quang · thanh vận · thiên chúa · **thiên
chúa giáo** · thiên vương tinh · thuỷ liễu · thân giáp · thạch bàn · thạch thán · **thần
chết** · **tin lành** · tiên sư · **trái đất** · trướng huỳnh · trường giang · trường xuân ·
tu lý · tuần duyên · tân kỳ · **tân ước** · tây dương · tây thiên · tô hạp · tùng lâm · tế
tân · **tết nguyên đán** · **tổ quốc** · tử phòng · u minh · vinh thăng · **việt ngữ** ·
vách quế · vân hà · vân hán · vân trình · võ miếu · **văn giáo** · **văn miếu** · **văn
nhân** · văn quan · **văn võ** · văn đức · **vũ công** · **vũ hội** · vũng tàu · vương bá ·
vạn an · vạn kiếp · vạn phúc · vầng ô · **vớ bở** · **xe tơ** · xuân hoá · xuân phong ·
xuân tình · xích thố · xương thịnh · yên bình · yên chi · yên hoa · yên hà · đinh điền ·
**đường luật** · **đường thi** · đại danh · đỉnh giáp non thần · **địa cầu** · đỗ vũ
Bold: everyday or dictionary-headword vocabulary by this auditor's reading, ~55 of 215.
The rest are Sino-Vietnamese literary terms, religious/astronomical names that Wiktionary
capitalizes by convention, and genuine place names (`hải phòng`, `vũng tàu`, `bồng sơn`).
**Narrowing does not fix this.** The plan's pre-decided narrowing — drop only when every
syllable is capitalized — was measured: drops fall 4,747 → 4,337, casualties fall only
215 → 147 (`mặt trời`, `trái đất`, `tổ quốc`-style two-cap forms stay dropped), and 410
proper nouns come back in, including junk like `bản mẫu:-vie-n-` and `con kde`. Rejected.
## 3. Corpus diff — old shipped vs new
| | |
|---|---|
| overlap | 21,320 |
| gained | 1,099 (`gân bò`, `rèn đúc`, `ích kỷ`, `đáng lý`, `thở hồng hộc`) |
| lost | **26,896** |
| — dropped as proper noun | 4,245 |
| — absent from the wiktionary branch | 22,651 |
| — other | 0 |
## 4. Playability
| metric | old | new |
|---|---|---|
| words | 48,216 | 22,419 |
| syllables | 6,676 | 5,485 |
| syllables that open a word | 5,049 | 4,052 |
| …with ≥2 continuations | 3,682 | 2,691 |
| dead-end syllables | 1,627 | 1,433 |
Bot-versus-bot, `internal/bot` real-corpus tests, 60 games each:
| | old | new |
|---|---|---|
| hard-vs-easy win rate (moves) | 98% (3.3) | 98% (3.5) |
| medium-vs-easy | 87% | 88% |
| hard-vs-medium (moves) | 72% (3.5) | 55% (2.9) |
| easy-vs-easy game length | 17.5 moves | **11.3 moves** |
| hard decision p95 | 5.2 ms | 1.1 ms |
Both tests pass on both databases. Easy-vs-easy games are a third shorter: the thinner
graph runs out of continuations sooner, which is the depth loss the plan predicted.
Hard-vs-medium falling to 55% means the graph gives the stronger bot fewer ways to trap.
## 5. Decision
Drop rule on the sample: **pass** (1 common word in 100).
Case casualties: **open — needs the owner's call**, because the options trade the
"wiktionary data only" decision against ~55 everyday words:
- **Accept** the 215 losses as measured.
- **Exception list**: a checked-in list of words to keep despite capitalization, curated
from the 215 above. Cause-aligned, small, but a hand-maintained artifact.
- **Read hongocduc for case evidence only**: recovers all 215 mechanically, but uses GPL
data as an input even though none of its words ship. Must be stated in `ATTRIBUTION.md`.
**Recorded decision (owner, 2026-09-08): abandon the capitalization rule.** Every
wiktionary-tagged word is kept regardless of case; proper nouns stay in the corpus as they
are in the shipped database today. The rule, its reject reason and `--report-drops` were
removed from the builder.
## 6. Final corpus, without the rule
```
accepted 26845 distinct words (sources: wiktionary)
```
| metric | old | new |
|---|---|---|
| words | 48,216 | 26,845 |
| syllables | 6,676 | 5,709 |
| syllables that open a word | 5,049 | 4,158 |
| …with ≥2 continuations | 3,682 | 2,787 |
| dead-end syllables | 1,627 | 1,551 |
| overlap / gained / lost | | 25,565 / 1,280 / 22,651 |
Every lost word is absent from the 2018 wiktionary branch; none is lost to a rule.
Bot-versus-bot, 60 games each: hard-vs-easy 95% (2.7 moves), medium-vs-easy 87%,
hard-vs-medium 63% (3.2 moves), easy-vs-easy **12.9 moves** (old: 17.5). Both real-corpus
tests pass.
plans/260908-1525-dictionary-corpus-switch/phase-01-start.md
-118
View File
@@ -1,118 +0,0 @@
---
phase: 1
title: "Phase 1: Read the merged list"
status: done
priority: P1
effort: "3h"
dependencies: []
---
# Phase 1: Read the merged list
## Overview
Teach `build-dictionary` a third input: undertheseanlp's merged JSONL. It selects words by
source membership and hands them to the normalization and filtering path that already
exists.
## Requirements
- Functional: read `{"text": "...", "source": ["hongocduc", ...]}` lines; keep a word when
its sources intersect the allowed set — **`wiktionary` only** by default.
- Functional: `hongocduc` and `tudientv` rows are skipped before any decision is made about
a word. They inform nothing.
- Functional: capitalization is not a filter. `Hà Nội` and `hà nội` are the same word and
land as the lowercase form, exactly as every other input mode already behaves.
- Functional: the `--min-words` default moves from 40,000 to 20,000. The new corpus is
~26,800 words; the old floor would reject it.
- Non-functional: streaming line-by-line. The file is 4.8 MB; nothing here needs to be
clever.
- Non-functional: everything downstream — `accept()`, syllable indexing, alias generation,
`verify()`, `meta` — is reused unchanged.
## Architecture
```
--merged <file> --sources wiktionary
│
▼ one row at a time
sources ∩ allowed ≠ ∅ ? ── no ──▶ skipped, never counted
│ yes
▼
accept() (NFC, lowercase, ≥2 syllables, alphabet, phonotactics)
│
▼
finish() (floor, aliases, atomic write, verify) ← shared with --words and --in
```
The `--sources` flag is kept general — a comma list validated against the three known
names — even though only `wiktionary` is passed, so a comparison build against another
branch is a flag away and never a code patch. Unknown names are an error, not a no-op.
`finish()` is new: the three input modes used to each carry their own copy of the floor
check, alias generation, write and verify. One copy is what keeps a fixture from drifting
into a different shape from the database production loads.
## Related Code Files
- Created: `server/cmd/build-dictionary/merged_list.go`
- Created: `server/cmd/build-dictionary/merged_list_test.go`
- Modified: `server/cmd/build-dictionary/main.go` — `--merged` and `--sources` flags,
mutually-exclusive input check, `finish()`, `sourceSpec.url` / `sourceSpec.extra`, meta
rows, package doc comment, `--min-words` default
- Unchanged: `server/cmd/build-dictionary/filter.go`
## Implementation Steps
1. Write `merged_list.go`: decode with `encoding/json` per line (`bufio.Scanner` with a
raised buffer — a scanner that silently truncates is a bad way to lose words), skip rows
with no allowed source, feed the rest to `accept()`.
2. Add `--merged <file>` and `--sources wiktionary` (the default) to `main.go`. Change the
`--min-words` default to 20,000 in the same edit; the fixture path passes its own floor
explicitly and is unaffected.
3. Dispatch in `run()`: exactly one of `--merged`, `--words`, `--in` may be given. Refuse
two rather than picking one.
4. Write `source_url`, `source_commit`, `sources_kept` and `sources_excluded` into `meta` —
provenance in the artifact, not only in a markdown file.
5. Tests in `merged_list_test.go`, table-driven, over hand-written JSONL fixtures:
- `Hà Nội` tagged wiktionary → kept as `hà nội`
- a `hongocduc`-only word and a `tudientv`-only word → absent from the output
- `học sinh` in all three, plus `Học sinh` → kept once
- a malformed line → a clear error naming the line number, not a skipped word
- `--sources hongocduc,wiktionary` → strictly more words; `tudientv` still absent
- meta rows as above
6. Run against the real file and record the counts.
## Result
```
rejected 2: contains a digit
rejected 147: contains punctuation
rejected 5335: fewer than 2 syllables
rejected 87: no Vietnamese letters
accepted 26845 distinct words (sources: wiktionary)
generated 1507 spelling aliases
```
The first cut of this phase also implemented a capitalization-based proper-noun drop
(`only ever capitalized` → rejected, `--report-drops` for audit). It reached the Phase 3
gate, was audited, and was removed by the owner's decision — see `plan.md` and
`audit-proper-noun-drops.md`. Nothing of it remains in the code.
## Success Criteria
- [x] `go test ./cmd/build-dictionary/` passes, new tests included.
- [x] Building from the real merged file accepts >20,000 words (26,845).
- [x] Ordinary-word probes present: `học sinh`, `bánh mì`, `xe đạp`, `giáo viên`, `hoa
hồng`, `tình yêu`, `nước mắm`, `mặt trời`, `trái đất`. `con người` and `cánh diều` are
not in the 2018 branch at all — expected, recorded.
- [x] `--sources wiktionary` (default) and `--sources hongocduc,wiktionary` both build, with
the first strictly smaller (26,845 vs 61,271 with no case rule).
- [x] No word in the output is reachable only through `hongocduc` or `tudientv`.
## Risk Assessment
**A JSON schema surprise.** The file is 8 years old and unversioned; a row could carry an
unexpected shape. Signal: decode errors on the real file. Response: the error names the
line — inspect it, and only then decide between a tolerant skip with a counted reason and
a hard failure. Silent skipping is not an option. (Observed: none; all 79,226 rows decode.)
plans/260908-1525-dictionary-corpus-switch/phase-02-pin-the-source.md
-87
View File
@@ -1,87 +0,0 @@
---
phase: 2
title: "Phase 2: Pin the source"
status: done
priority: P1
effort: "2h"
dependencies: [1]
---
# Phase 2: Pin the source
## Overview
Point the build machinery at the new file: pinned by commit and checksum, fetched by
`make fetch-dict` and by the Docker dictionary stage, with the existing agreement test
still guarding both.
## Requirements
- Functional: `make fetch-dict` downloads the pinned JSONL and verifies its SHA-256;
`make dict` builds `data/noitu.db` from it.
- Functional: the Docker `dict` stage does the same, and its fixture escape hatch
(`FIXTURE_DICT=1`) keeps working untouched.
- Functional: the fetched source file is gitignored — we distribute the derived database,
never a copy of their file.
- Non-functional: `DICT_URL` / `DICT_SHA256` keep their names so
`web/tests/dictionary-source.test.js` continues to guard Makefile/Dockerfile agreement
with no change to the test.
## Architecture
The pin is a commit-addressed raw URL, which is immutable — unlike `master`, where the
URL and the checksum could silently diverge:
```make
DICT_URL := https://raw.githubusercontent.com/undertheseanlp/dictionary/2c078cfc373b06e2980d324ce1d7bd13740c3319/dictionary/words.txt
DICT_SHA256 := 4c3e0e6117e4bdfa97731e135c3d4a05881889909267394a8de8d88ef79f13f0
DICT_SRC := data/undertheseanlp-words.jsonl
```
`fetch-dict` and `verify-dict` keep their current shape — `curl` then `sha256sum -c` —
because that is already the right shape; only the size in the help text changes.
## Related Code Files
- Modify: `Makefile` — `DICT_URL`, `DICT_SHA256`, `DICT_SRC`, the `dict` target's flags
(`--merged` in place of `--in`), and the `help` text's "~179 MB"
- Modify: `Dockerfile` — `ARG DICT_URL`, `ARG DICT_SHA256`, the `curl`/`sha256sum` lines,
the `build-dictionary` invocation, and the `FIXTURE_DICT` comment's 179 MB reference
- Modify: `.gitignore` — ignore `data/undertheseanlp-words.jsonl` (`data/*.db` already
covers the derived and old source databases)
- Verify unchanged: `web/tests/dictionary-source.test.js`, `.github/workflows/ci.yml`
## Implementation Steps
1. Update the three Makefile variables and the `dict` target to pass `--merged $(DICT_SRC)`.
`--sources` is left at its `wiktionary` default so the Makefile and Dockerfile carry
one fewer thing to keep in agreement.
2. Reword `help`, `fetch-dict` and the `$(DICT_SRC)` guard: the download is ~4.8 MB now,
and saying "179 MB" would be the kind of stale comment that outlives three refactors.
3. Mirror all of it in the Dockerfile stage. Keep `FIXTURE_DICT=1` on `--words` — the
image smoke test must not start needing a network.
4. Add the source file to `.gitignore` and confirm `git status` is clean after a fetch.
5. Run `make fetch-dict && make verify-dict && make dict` end to end from a clean `data/`.
6. Run `npm test` in `web/` — `dictionary-source.test.js` should pass untouched. If it
needs editing, the variable names were changed unnecessarily; put them back.
7. Build the image with and without `FIXTURE_DICT=1` and confirm both produce a database.
## Success Criteria
- [ ] `make fetch-dict` pulls 4,813,111 bytes and the checksum verifies.
- [ ] `make dict` produces `data/noitu.db` with the Phase 1 word count.
- [ ] `web/tests/dictionary-source.test.js` passes with no edits to it.
- [ ] `docker build` succeeds both with `FIXTURE_DICT=1` and without.
- [ ] `git status` is clean after a fetch — no source file staged, ever.
- [ ] No file in the repository still claims a 179 MB dictionary download.
## Risk Assessment
**`raw.githubusercontent.com` is not an archive.** An 8-year-dormant repository could be
deleted or renamed, and then `make fetch-dict` fails for everyone. Signal: a 404 on fetch.
Response: the checksum makes any mirror verifiable, so host a copy as a release asset on
this repository and re-pin to it. Worth doing pre-emptively if the build ever gates CI.
**The Docker stage and the Makefile drift.** Signal: the agreement test fails. Response:
that is the test doing its job — fix the file that is behind, and do not weaken the test to
match.
plans/260908-1525-dictionary-corpus-switch/phase-03-audit-the-corpus.md
-110
View File
@@ -1,110 +0,0 @@
---
phase: 3
title: "Phase 3: Audit the corpus"
status: done
priority: P1
effort: "2h"
dependencies: [1, 2]
---
# Phase 3: Audit the corpus
> **Outcome.** Executed 2026-09-08; see [`audit-proper-noun-drops.md`](./audit-proper-noun-drops.md).
> The drop rule passed its sample gate (1 common word in 100) but cost 215 words under
> wiktionary-only case evidence, including everyday vocabulary. Narrowing was measured and
> rejected. **The owner abandoned the rule**: every word is kept regardless of case. Final
> corpus 26,845 words; playability and bot results recorded in the audit note.
## Overview
The gate. A rule that removes thousands of words on a capitalization heuristic has to be
looked at before it ships. This phase samples what was dropped, measures what the corpus
gained and lost, and confirms the game graph is not worse.
## Requirements
- Functional: a hand audit of 100 randomly sampled proper-noun drops, recorded in the
plan directory with the verdict per entry.
- Functional: a measured diff against the current `data/noitu.db` — gained, lost, and the
reason for each loss bucket.
- Functional: a playability comparison on the graph the game actually walks.
- Non-functional: every number in this phase is reproducible from a command written down
in the audit note, not from a one-off shell line nobody kept.
## Architecture
Three measurements, each against the freshly built database and the current one:
1. **Drop audit.** Sample 100 of the dropped keys. Each is *proper noun* (correct),
*common word* (a false positive, the thing we are looking for), or *unclear*. The rule
passes at ≤2 common words in 100; that tolerance is the difference between a filter and
a corpus edit.
2. **Corpus diff.** Overlap, gained, lost. Split the lost into: dropped as proper nouns,
absent from the wiktionary branch. The report measured 4,335 / 22,651 of 26,986 lost;
this re-measures from the shipped build. Separately count the **271 case casualties**
— drops that have a lowercase form in `hongocduc` or `tudientv` — and list them in
full; that list is what the pass/narrow/exception decision is made on.
3. **Playability.** Words, syllables, syllables that can open a word, how many have ≥2
continuations, and dead-end syllables — the last of these being what decides whether
the game hands somebody an unanswerable position. Current: 48,216 / 6,676 / 5,049 /
3,682 / 1,627. Expected: 22,310 / 5,484 / 4,050 / 2,686 / 1,434. The corpus is smaller
by design, so the question is not "is it bigger" but "does the game still run": add
bot-versus-bot game lengths from `server/internal/bot`'s real-corpus tests on both
databases.
## Related Code Files
- Create: `plans/260908-1525-dictionary-corpus-switch/audit-proper-noun-drops.md` —
the sample, the verdicts, the commands that produced them
- Read only: `data/noitu.db` (current, before regeneration), the newly built database
## Implementation Steps
1. Build the new database to a scratch path, leaving the shipped one in place for
comparison. Do not overwrite `data/noitu.db` in this phase.
2. Emit the dropped keys from the builder — a `--report-drops <file>` flag, or a temporary
local patch if the flag would otherwise have no user. Prefer the flag: the next person
to change this rule will want it too.
3. Sample 100 with a fixed seed, write them into the audit note, and judge each one by
hand. Vietnamese place and person names are the expected content; anything that reads
as an ordinary word is a false positive and gets called one.
4. Run the corpus diff and the playability comparison; put both tables in the audit note.
5. Build `--sources hongocduc,wiktionary` to a second scratch path purely to enumerate the
case casualties (words dropped under `wiktionary` alone but kept when hongocduc's
lowercase forms are visible). Nothing from that build ships.
6. Decide: pass, narrow, exception list, or abandon the rule. Record the decision and its
reason in the audit note — and if the rule changes, Phase 1's tests change with it.
## Success Criteria
- [ ] The audit note exists, with 100 judged entries and the commands that produced them.
- [ ] ≤2 of 100 sampled drops are ordinary words, the 271 known case casualties aside —
those are listed in full and judged as a group.
- [ ] The new corpus has >20,000 words.
- [ ] The five graph numbers and bot-game lengths are recorded for both databases; bot
games on the new graph complete without the engine running out of moves earlier
than on today's.
- [ ] The loss buckets are quantified, not estimated.
- [ ] A pass/narrow/exception-list/abandon decision is written down with its reason.
## Risk Assessment
**The audit passes on a sample and the corpus is still wrong in the tail.** 100 of several
thousand is a sample, not a proof. Signal: players disputing rejected words after release.
Response: the drop list is a file — a disputed word can be checked against it in seconds,
and a per-word exception list is a small change on top of this design.
**The audit fails and the phase becomes a redesign.** Signal: >2 common words in 100.
Response: apply Phase 1's pre-decided narrowing (every syllable capitalized) or the
exception list, re-audit once. If it fails again, ship without the drop — the proper-noun
problem then stays exactly as bad as it is today rather than getting worse.
**The corpus is too thin in play.** 22,310 words is less than half of today's. Signal: bot
games noticeably shorter, or the dead-end rate per move up rather than down. Response: this
is the trigger for the documented upgrade path — the 2026-09-01 viwiktionary dump (31,637
words, same license) as a second `--words` source — not for re-admitting GPL data.
**Playability regresses in a way these five numbers do not capture.** Signal: aggregate
counts look fine but bot games end oddly short or long. Response: `server/internal/bot`'s
real-corpus tests run bot-versus-bot games on the shipped database; run them before
accepting the phase, since they exercise the graph rather than counting it.
plans/260908-1525-dictionary-corpus-switch/phase-04-update-the-attribution.md
-116
View File
@@ -1,116 +0,0 @@
---
phase: 4
title: "Phase 4: Update the attribution"
status: done
priority: P1
effort: "1h"
dependencies: [3]
---
# Phase 4: Update the attribution
> **Outcome.** Done 2026-09-08. `data/LICENSE` untouched; `NOTICE` section 2 names the new
> upstream, `data/ATTRIBUTION.md` rewritten (eight modifications, both attribution links,
> both exclusions). Also updated, beyond the plan's list: the frontend attribution footer
> (`AttributionFooter.svelte`, `i18n/vi.js`) and its e2e assertion now credit Wiktionary
> tiếng Việt instead of minhqnd — the user-visible half of the CC BY-SA obligation.
## Overview
The data license does not change — the wiktionary branch is CC BY-SA 4.0, the same license
`noitu.db` already carries — but everything that names the source does. This phase rewrites
the attribution and notice text and regenerates the shipped database in one commit, so the
attribution never describes a database other than the one in the tree.
## Requirements
- Functional: `data/LICENSE` is untouched — still the CC BY-SA 4.0 text.
- Functional: `NOTICE` section 2 names the new upstream, keeps CC BY-SA 4.0, and keeps
section 1 (Apache-2.0 for code) byte-identical.
- Functional: `data/ATTRIBUTION.md` records the new source, the branch actually used, the
two branches excluded and why, and the full list of modifications.
- Functional: the regenerated `data/noitu.db` carries matching `meta` provenance.
- Non-functional: no overclaiming. The branch is a 2018 scrape of vi.wiktionary.org made by
a third party; say that, and attribute Wiktionary's contributors as CC BY-SA requires.
## Architecture
The dual-license structure in `NOTICE` is correct and stays; only the source lines of its
second section change:
```
1. SOURCE CODE — Apache-2.0 (unchanged: server/, tools/, web/, proto/)
2. DICTIONARY DATA — CC BY-SA 4.0 (unchanged license, new upstream)
data/noitu.db, data/LICENSE, data/ATTRIBUTION.md
```
The attribution chain is now two links instead of an opaque aggregate: vi.wiktionary.org
contributors (CC BY-SA) → `undertheseanlp/dictionary`, branch data `wiktionary`, commit
`2c078cf` → this project. Both links are named.
The modifications list stays at eight: source selection replaces the old language filter
as entry 1, which the license requires us to declare as a change.
## Related Code Files
- Modify: `NOTICE` — section 2's upstream line, affected-artifacts line (`dictionary.db`
is no longer the source), and the closing paragraph about build artifacts
- Modify: `data/ATTRIBUTION.md` — source table, branch selection, exclusions, modifications
9 and 10, the reproduce block
- Modify: `README.md` — dictionary-source lines (`~179 MB`, `dictionary.db`, `minhqnd`,
the manual `curl` + `--in` example)
- Modify: `docs/deployment.md` — the 179 MB builder-stage sentence
- Verify unchanged: `data/LICENSE`, `.github/workflows/ci.yml` required-files check
- Regenerate: `data/noitu.db`
## Implementation Steps
1. Rewrite `NOTICE` section 2's source lines: upstream `undertheseanlp/dictionary` at
commit `2c078cf`, wiktionary data only; affected artifact `data/noitu.db`. Leave the
share-alike paragraph as is — it is still true.
2. Rewrite `data/ATTRIBUTION.md`:
- source is `undertheseanlp/dictionary`, file `dictionary/words.txt`, commit `2c078cf`
- rows used: those tagged `wiktionary` — a 2018-12-10 scrape of vi.wiktionary.org,
whose text is CC BY-SA 4.0 / GFDL by its contributors
- `hongocduc` **excluded**: GNU GPL per the branch README; using it would relicense this
database to GPLv3
- `tudientv` **excluded**: its README declares copyright *"Chưa rõ"* and names
Soha/Vietlex (Hoàng Phê) as its primary source. Say it plainly so nobody reopens it.
- modifications renumbered: **1. source selection** replaces the language filter (the new
file is Vietnamese-only); the other seven carry over, with normalization stating that
capitalization removes nothing — the proper-noun drop was abandoned at the Phase 3 gate
- reproduce block: `make fetch-dict` is ~4.8 MB now
3. Grep the repository for `minhqnd`, `dictionary.db`, `179` and `--in` and fix every
survivor outside `plans/` — the reports are dated records and stay as written.
4. Regenerate `data/noitu.db` and confirm its `meta` rows agree with `ATTRIBUTION.md`:
source URL, commit, license, `sources_kept = wiktionary`,
`sources_excluded = hongocduc,tudientv`.
5. Run the CI required-files check locally, or read it and confirm by hand.
## Success Criteria
- [ ] `data/LICENSE` has no diff.
- [ ] `NOTICE` section 2 names the new upstream and still says CC BY-SA 4.0; section 1 is
byte-identical.
- [ ] `data/ATTRIBUTION.md` lists eight modifications, names the branch used and the two
excluded with reasons, and credits vi.wiktionary.org contributors.
- [ ] `meta` in the built database matches the attribution file.
- [ ] No file outside `plans/` names minhqnd, `dictionary.db` as a source, or a 179 MB
download.
- [ ] CI's required-files check passes.
## Risk Assessment
**A stale source claim survives somewhere.** A README line or Docker comment naming the old
upstream is an attribution misstatement, not a typo. Signal: the grep in step 3 finds
something after the phase is called done. Response: grep is a step in the phase for that
reason; run it last, not first.
**Section 1 gets damaged while section 2 is edited.** The code's Apache-2.0 grant is not
in scope and must come out byte-identical. Signal: a diff touching section 1. Response:
review the `NOTICE` diff before committing.
**The attribution credits the wrong party.** CC BY-SA attribution belongs to Wiktionary's
contributors; undertheseanlp is the intermediary that scraped and redistributed. Signal:
an `ATTRIBUTION.md` that names only the GitHub repository. Response: step 2 names both
links of the chain explicitly.
plans/260908-1525-dictionary-corpus-switch/phase-05-retire-the-sqlite-path.md
-96
View File
@@ -1,96 +0,0 @@
---
phase: 5
title: "Phase 5: Retire the SQLite path"
status: done
priority: P2
effort: "1h"
dependencies: [4]
---
# Phase 5: Retire the SQLite path
> **Outcome.** Done 2026-09-08. `--in`, `--table`, `--word-col`, `--lang-col`, `--lang`,
> `resolveSource`, `validateSource`, `listTables`, `listColumns`, `pickColumn`, `extract`
> and `quoteIdent` removed; `meta` no longer writes `source_word_column` /
> `source_lang_column` (no consumer existed). The four auto-detection tests went; the
> output-shape tests now build their fixture through `--merged`. `--help` lists `--merged`,
> `--sources`, `--words`, `--out`, `--max-syllables`, `--min-words` and nothing else.
## Overview
Delete the input mode nothing uses any more: reading a source SQLite database, with the
table and column auto-detection that existed because the upstream schema was not ours to
rely on. Two input modes for one source is dead weight, and dead weight in a build tool is
where stale assumptions hide.
## Requirements
- Functional: `--in`, `--table`, `--word-col`, `--lang-col` and `--lang` are gone, along
with the schema resolution behind them.
- Functional: `--merged` (the corpus) and `--words` (the fixture) remain. The fixture path
stays because the e2e suite and the Docker smoke build depend on it.
- Non-functional: the package doc comment describes what the tool now does, not what it
used to.
- Non-functional: `modernc.org/sqlite` stays a dependency — it writes the output database.
## Architecture
What `run()` dispatches on collapses from three modes to two:
```
--merged <file> the corpus: undertheseanlp JSONL (default path)
--words <file> the fixture: one word per line (e2e, Docker smoke build)
```
`resolveSource` and its auto-detection tests go with `--in`. They were never about the
game's rules — they were insurance against an upstream schema we did not control, and we
no longer read that upstream.
## Related Code Files
- Modify: `server/cmd/build-dictionary/main.go` — remove `--in` and the four schema flags,
`resolveSource`, the SQLite read path, and the `config` fields behind them; rewrite the
package doc comment and the usage example
- Modify: `server/cmd/build-dictionary/main_test.go` — remove the source-resolution tests;
keep and extend the ones about output shape
- Modify: `Makefile` — drop any remaining `--in` reference and 179 MB wording from `help`
- Verify: `Dockerfile` no longer mentions `--in`
## Implementation Steps
1. Remove the flags, the `config` fields, `resolveSource` and the SQLite read path.
2. Delete the tests that exist only to cover auto-detection. Do not delete tests covering
output shape, `verify()`, or the reject-reason counts — those still describe behavior.
3. Rewrite the package doc comment: the input is a 4.8 MB JSONL wordlist with source
membership and capitalization, of which only wiktionary-tagged rows are read; the
output is the game's syllable-indexed database.
4. `go build ./... && go vet ./... && go test ./...` from `server/`.
5. Grep for `--in`, `dictionary.db`, `resolveSource` and `lang_code` outside `plans/`;
nothing should survive except in the dated reports.
6. `make dict` once more from a clean `data/` to prove the surviving path is the one that
works.
## Success Criteria
- [ ] `go build`, `go vet` and `go test ./...` are clean from `server/`.
- [ ] `build-dictionary --help` lists `--merged` and `--words` and nothing about tables,
columns or language codes.
- [ ] `make dict` and `make fixture-dict` both work.
- [ ] The e2e suite still builds its fixture and passes.
- [ ] No reference to `--in`, `resolveSource` or `lang_code` survives outside `plans/`.
## Risk Assessment
**Deleting the wrong tests.** The auto-detection tests and the output-shape tests live in
the same file. Signal: coverage of `verify()` or the reject counters disappears with the
deletion. Response: read `main_test.go` in full before cutting, and delete by test
function rather than by line range.
**Somebody wants the SQLite path back.** A future source could ship SQLite again. Signal:
a future plan proposing it. Response: git remembers, and re-adding a reader for a schema
we would then actually control is a smaller job than maintaining auto-detection for a
schema nobody reads. Deleting it is the cheaper bet.
**The fixture path is quietly coupled to something removed.** Signal: `make fixture-dict`
or the e2e build fails after the cut. Response: step 6 builds both paths; run it before
calling the phase done, not after.
plans/260908-1525-dictionary-corpus-switch/plan.md
-155
View File
@@ -1,155 +0,0 @@
---
title: "Dictionary corpus switch"
description: "Replace the 179 MB minhqnd aggregate with the wiktionary branch of undertheseanlp/dictionary only (hongocduc and tudientv excluded) and keep data/noitu.db on CC BY-SA 4.0"
status: completed
priority: P1
effort: "~1d"
tags: [dictionary, data, licensing, build]
created: 2026-09-08
updated: 2026-09-08
blockedBy: []
blocks: []
---
# Dictionary corpus switch
## Overview
`data/noitu.db` was derived from `minhqnd/dictionary` v2.0.0: a 179 MB SQLite aggregate of
five upstreams, redistributed as CC BY-SA 4.0. Its license chain does not close — three of
its five inputs are GPL (FVDP, tudientv, "Vietnamese Explanatory Dictionary") and GPL does
not permit relicensing to CC BY-SA.
This plan switches the corpus to the **`wiktionary` source inside `undertheseanlp/dictionary`**
— a single 4.8 MB JSONL file carrying, per word, which of three source dictionaries contain
it. Only rows whose `source` list contains `wiktionary` are read. `hongocduc` (GPL) is
excluded so the data license does not change; `tudientv` is excluded because its own README
declares copyright *"Chưa rõ"* and names Soha/Vietlex (Hoàng Phê) as its primary source.
`data/noitu.db` **stays CC BY-SA 4.0**: the wiktionary branch is a 2018-12-10 scrape of
vi.wiktionary.org, whose text is CC BY-SA.
Evidence, all measured through the real `build-dictionary` filter:
`plans/reports/research-260908-1507-undertheseanlp-dictionary.md` (branch licensing),
`plans/reports/research-260908-1529-viwiktionary-dump-measured.md` (yields, alternatives)
and [`audit-proper-noun-drops.md`](./audit-proper-noun-drops.md) (the Phase 3 gate).
## What changes, in numbers
| | before | after |
|---|---|---|
| words | 48,216 | **26,845** |
| syllables | 6,676 | 5,709 |
| syllables that can open a word | 5,049 | 4,158 |
| …with ≥2 continuations | 3,682 | 2,787 |
| dead-end syllables | 1,627 | 1,551 |
| source download | 179 MB SQLite | 4.8 MB JSONL |
| data license | CC BY-SA 4.0 | **CC BY-SA 4.0 (unchanged)** |
| `--min-words` floor | 40,000 | **20,000** |
**This is a deliberate 44% cut.** 1,280 words gained, 22,651 lost — every one of them
simply absent from a 2018 wiktionary scrape (`bánh đúc`, `lò vi sóng`, `dân làng`, `thót
tim`). Bot-versus-bot games run about a quarter shorter on the thinner graph (easy-vs-easy
12.9 moves vs 17.5). This loss is accepted here; the upgrade path is written down below.
## Decisions taken
- **Only `wiktionary` rows are read.** `hongocduc` and `tudientv` inform nothing. That is
what keeps the license CC BY-SA 4.0 and what makes "we do not touch that data" true
without qualification.
- **Capitalization is not a filter.** The original design dropped words that appear only
ever capitalized, as proper nouns. It was built, run and audited at the Phase 3 gate: the
rule was precise on a 100-word sample (1 common word) but, with wiktionary-only case
evidence, it also removed 215 words including `mặt trời`, `trái đất`, `tổ quốc`, `dường
như`, `phật giáo`. Narrowing was measured and did not help. **The owner chose to keep
every word regardless of case.** Proper nouns like `hà nội` remain playable, exactly as
they are today. The mechanism was removed from the builder rather than left behind a flag.
- **Rejected alternatives, so nobody reopens them without new evidence:**
- `hongocduc + wiktionary` (61,271 words) — forces `noitu.db` to GPLv3. Rejected to keep
CC BY-SA 4.0.
- the 2026-09-01 viwiktionary dump (36,200 words without a case rule, same license,
monthly pin) — the 2018 branch is a near-subset of it. Rejected for now in favour of the
simpler single-file pin; **this is the documented upgrade path** if the 27k corpus
proves too thin in play. See the 15:29 report.
- kaikki.org enwiktionary Vietnamese — unpinnable rolling file, deprecated by its host.
- **The SQLite input path is deleted, not kept "just in case."** The fixture path
(`--words`) stays because the e2e suite and the Docker smoke build use it.
- **The build floor drops to 20,000.** The old 40,000 default would reject the new corpus;
it moved once, in `main.go`.
## The pin
```
URL https://raw.githubusercontent.com/undertheseanlp/dictionary/2c078cfc373b06e2980d324ce1d7bd13740c3319/dictionary/words.txt
SHA256 4c3e0e6117e4bdfa97731e135c3d4a05881889909267394a8de8d88ef79f13f0
Size 4,813,111 bytes — 79,226 JSONL rows, 32,484 of them tagged wiktionary
Commit 2c078cf (2018-12-10, the repository's last)
```
Pinned by commit rather than branch: `raw.githubusercontent.com/<repo>/<sha>/` is
immutable, so the URL and the checksum cannot disagree later. The variable names
`DICT_URL` / `DICT_SHA256` were kept so `web/tests/dictionary-source.test.js` needed no
change — only their values moved.
## Phases
| # | Phase | Status |
|---|-------|--------|
| 1 | [Read the merged list](./phase-01-start.md) | Done |
| 2 | [Pin the source](./phase-02-pin-the-source.md) | Done |
| 3 | [Audit the corpus](./phase-03-audit-the-corpus.md) | Done — gate sent the case rule back; rule abandoned |
| 4 | [Update the attribution](./phase-04-update-the-attribution.md) | Done |
| 5 | [Retire the SQLite path](./phase-05-retire-the-sqlite-path.md) | Done |
All five phases are uncommitted in one working tree; they should land together so the
attribution never describes a database other than the one the build produces.
## Success criteria
- [x] `make fetch-dict && make dict` builds `data/noitu.db` from a 4.8 MB pinned file with
no 179 MB download.
- [x] The shipped database has **more than 20,000 words** and **no word that reaches it
through `hongocduc` or `tudientv` alone**.
- [x] `meta` records the source URL, commit, sources kept and sources excluded.
- [x] Playability is measured and recorded: syllables, openers, continuations, dead ends,
and bot-versus-bot game lengths on the new graph versus the old.
- [x] `data/LICENSE` is unchanged; `NOTICE` and `data/ATTRIBUTION.md` name the new source,
its commit and license, and record why the two other branches are excluded. The
frontend attribution footer credits Wiktionary tiếng Việt.
- [x] `go vet ./... && go test ./...` (incl. `-race` on the touched packages), `npm run
check && npm test`, and the Docker image (`noitu:ci` with the CI file checks, plus
`FIXTURE_DICT=1`) are green; the fixture build path is untouched.
- [ ] The Playwright e2e suite is green. **Not verified locally**: Playwright's Chromium
download timed out twice on this machine (2026-09-08), so every test failed at browser
launch, not on game behaviour. The one e2e assertion this plan changes (the attribution
footer's link text and href in `web/e2e/bot-game.spec.js`) was updated by hand. Verify
in CI or after `npx playwright install chromium` succeeds.
- [x] `--in` and its schema auto-detection are gone, and no file outside `plans/` still
mentions minhqnd, `dictionary.db`, `--in` or a 179 MB download.
## Licence version — decided after review
The owner asked whether the data could move to Apache-2.0 to simplify the repo. It cannot:
the words are Wiktionary contributors' text under a share-alike licence, and neither
undertheseanlp nor this project can relicense them. The owner then chose to **keep the
licence the source text actually carried, CC BY-SA 3.0 Unported**, rather than upgrade the
derivative to 4.0 under the later-version clause. `data/LICENSE` is now the BY-SA 3.0 legal
code; `NOTICE`, `ATTRIBUTION.md`, README, Dockerfile, deployment doc, the builder's meta
string and the in-game footer all say 3.0. The "unchanged" wording above predates this.
## Review
Code review (2026-09-08) returned DONE_WITH_CONCERNS; every finding was applied: the
builder's copy of the upstream commit is now guarded by `web/tests/dictionary-source.test.js`
alongside the Makefile/Dockerfile pin; fixture builds record "no upstream data" instead of
inheriting the CC BY-SA string; `ATTRIBUTION.md` states the 2018 snapshot was CC BY-SA 3.0
and why the derived database is 4.0; dead `contains()`, redundant sorts, the off-by-one
scanner line number, stale "48k" comments, the Makefile `curl` flags and the Dockerfile
download name were fixed; input-selection and fixture-licence tests were added.
## Open questions
- Is 26,845 words enough for the game to feel playable? Bot games are ~25% shorter; only
real play can say whether that is felt. If not, the 2026 viwiktionary dump is the next
step, not a return to GPL data.
<!-- slug: dictionary-corpus-switch -->
plans/260908-1653-kaikki-viwiktionary-corpus/measurement.md
-97
View File
@@ -1,97 +0,0 @@
# Measurement: first kaikki build vs the undertheseanlp corpus
Date: 2026-09-08. Phase 3 of `plan.md`. The pre-switch `data/noitu.db` (26,845 words) was
copied aside before the first kaikki build and is the "old" side of every table.
## The file measured
```
source_url https://kaikki.org/viwiktionary/Ti%E1%BA%BFng%20Vi%E1%BB%87t/kaikki.org-dictionary-Ti%E1%BA%BFngVi%E1%BB%87t.jsonl
source_sha256 51ddc2fbd73cd7e2a468ccff3c31200928e7200cc58622f528ba3eb84c7a49f0
source_rows 44564
source_fetched_at 2026-09-08T10:04:08Z (kaikki's export of 2026-09-06, dump 2026-09-01)
size 62,288,004 bytes; longest line 61,552 bytes
```
Build log:
```
rejected 4: contains a digit
rejected 136: contains punctuation
rejected 8348: fewer than 2 syllables
rejected 160: no Vietnamese letters
parts of speech: noun 16219, verb 9355, adj 6698, name 5180, unknown 4652, adv 1024,
phrase 392, proverb 322, character 198, intj 175, pron 100, conj 70, prep 48, num 45,
abbrev 38, particle 18, romanization 14, prefix 13, suffix 2, det 1
accepted 34813 distinct words
generated 1664 spelling aliases (552 skipped as ambiguous or already real words)
```
## Commands
```sh
cp data/noitu.db /tmp/old.db # before the switch
make fetch-dict && make dict # or the raw curl + go run lines in README
cd server && go test ./internal/bot/ -run RealCorpus -v -count=1 # once per database at data/noitu.db
# graph numbers and diff: sqlite, see the queries below
```
```sql
SELECT COUNT(*) FROM words;
SELECT COUNT(*) FROM syllables;
SELECT COUNT(DISTINCT first) FROM words;
SELECT COUNT(*) FROM (SELECT first FROM words GROUP BY first HAVING COUNT(*) >= 2);
SELECT COUNT(*) FROM syllables WHERE out_degree = 0;
```
## Playability
| metric | old (undertheseanlp) | new (kaikki) |
|---|---|---|
| words | 26,845 | **34,813** |
| syllables | 5,709 | 6,081 |
| syllables that open a word | 4,158 | 4,487 |
| …with ≥2 continuations | 2,787 | 3,163 |
| dead-end syllables | 1,551 | 1,594 (27.2% → 26.2% of syllables) |
Bot-versus-bot, 60 games each, same run:
| | old | new |
|---|---|---|
| hard-vs-easy win rate (moves) | 98% (3.6) | 97% (3.2) |
| medium-vs-easy | 87% | 95% |
| hard-vs-medium (moves) | 57% (3.1) | 67% (3.1) |
| easy-vs-easy game length | 14.4 moves | 14.4 moves |
| hard decision p95 | 1.5 ms | 3.1 ms |
Both real-corpus tests pass on both databases. Game length is unchanged; the stronger bots
win more often on the denser graph, which is the expected direction.
## Corpus diff
| | |
|---|---|
| shared | 25,392 |
| gained | **9,421** — `công nghiệp hóa`, `bóng râm`, `bội thu`, `đồng dao`, `bỏ ngỏ`, `đắt hàng`, `ấn độ giáo`, `chạy làng`, `béo nung núc`, `nhìn muốn rụng trứng` |
| lost | **1,453** — see below |
### The 1,453 lost words
Sample of 25 (seed 7): `trịnh tuệ`, `tuân khanh`, `qua quít`, `cúc pha`, `mành mành`,
`cây bài chặt`, `trảm phong`, `kéo cánh`, `ngọt lự`, `rưng rức`, `làu nhàu`, `trong lúc`,
`khí hư`, `trần ích tắc`, `nháo nhác`, `trần cảnh`, `óc ách`, `lườm lườm`, `hỗn luân`,
`trụi lủi`, `tục tác`, `lảu nhảu`, `phát sầu`, `hãng hàng không quốc gia việt nam`, `trói ké`.
Two kinds. Roughly a fifth are historical person names (`trần ích tắc`, `trần cảnh`, `trịnh
tuệ`) whose pages have since been deleted or moved. The rest are real vocabulary, heavily
reduplicatives (`rưng rức`, `nháo nhác`, `óc ách`, `làu nhàu`, `lườm lườm`), that the 2018
scrape had and the current wiktextract export does not. Most likely those pages still exist
but in a markup the `vi` extractor does not yet parse — the raw 2026 dump has 36,200
words to kaikki's 34,813, a 1,387-word gap of the same order. **Not acted on:** the plan
does not merge sources, and the net is +7,968 words. If the reduplicatives matter, the
option is a second `--words` input reduced from the raw dump, as the previous plan's
research described.
## Verdict
Every number above today's; nothing regresses. Corpus accepted as built.
plans/260908-1653-kaikki-viwiktionary-corpus/phase-01-start.md
-118
View File
@@ -1,118 +0,0 @@
---
phase: 1
title: "Phase 1: Read the kaikki file"
status: completed
priority: P1
effort: "3h"
dependencies: []
---
# Phase 1: Read the kaikki file
## Overview
Replace the undertheseanlp reader in `build-dictionary` with one for kaikki's wiktextract
JSONL, hashing the input as it streams so the database records exactly which bytes it was
built from.
## Requirements
- Functional: `--kaikki <file>` reads one JSON object per line, keeps rows whose
`lang_code` is `vi`, and feeds `word` to the existing `accept()`.
- Functional: rows with any other `lang_code` are counted and rejected, not silently
skipped — the file is vi-only today, and a change in that would be worth seeing in the log.
- Functional: `pos` is tallied per value and printed with the reject counts. It never filters.
- Functional: the SHA-256 of the file and the row count are computed during the single
streaming pass and written to `meta` as `source_sha256` and `source_rows`, with
`source_fetched_at` taken from the file's modification time.
- Functional: `--merged` and `--sources` are removed along with `merged_list.go` and its
tests. Exactly one of `--kaikki` / `--words` must be given, same rule as today.
- Functional: the `--min-words` default moves from 20,000 to 30,000.
- Non-functional: a malformed line is an error naming the line, as today. Lines can be
long — a row carries every sense and translation — so the scanner buffer must allow
several megabytes; measure the longest line on the real file and set the cap above it
with headroom, or switch to `bufio.Reader.ReadBytes('\n')` which has no cap.
- Non-functional: `finish()`, `accept()`, aliases, `write`, `verify` unchanged.
## Architecture
```
--kaikki <file>
│ one line at a time, bytes also fed to sha256.New()
▼
json.Unmarshal → {word, lang_code, pos}
│
├─ lang_code != "vi" → rejects["not Vietnamese-language entry"]++
▼
accept(word) (NFC, lowercase, ≥2 syllables, alphabet, phonotactics)
▼
finish() (floor 30,000, aliases, atomic write, verify)
```
Decode only the three fields via a struct; `encoding/json` ignores the rest, so the 62 MB
of senses and translations cost I/O but not memory.
`sourceSpec.extra` already exists for mode-specific meta rows; the kaikki provenance uses it:
```
source_url the kaikki URL (constant kaikkiSourceURL)
source_sha256 hex of the streamed bytes
source_rows lines decoded
source_fetched_at file mtime, RFC 3339 UTC
source_license CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
```
## Related Code Files
- Create: `server/cmd/build-dictionary/kaikki_list.go`
- Create: `server/cmd/build-dictionary/kaikki_list_test.go`
- Delete: `server/cmd/build-dictionary/merged_list.go`, `merged_list_test.go`
- Modify: `server/cmd/build-dictionary/main.go` — flags, dispatch, doc comment, `--min-words`
default, `mergedProvenance` call site
- Modify: `server/cmd/build-dictionary/main_test.go` — `fixtureSource`/`defaultRows` emit
kaikki-shaped rows (`{"word": ..., "pos": ..., "lang_code": "vi"}`); the "excluded source"
row becomes a `lang_code: "en"` row
## Implementation Steps
1. Write `kaikki_list.go`: `kaikkiRow{Word, Pos, LangCode}`, `readKaikkiList(path, maxSyllables)`
returning words, rejects, a `pos` tally, and a `kaikkiProvenance` (sha256, rows, mtime).
Wrap the file in `io.TeeReader` into `sha256.New()` so hashing is free.
2. Add `rejectNotVietnamese rejectReason = "not a Vietnamese-language entry"` to `filter.go`.
3. In `main.go`: replace `merged`/`sources` with `kaikki` in `config`, flags and `run()`;
`--min-words` default 30,000; `runFromKaikkiList` logs rejects, the POS tally
(sorted, one line) and the accepted count, then calls `finish()` with the provenance spec.
4. Delete the merged reader and its tests. Rewrite `main_test.go` fixtures to kaikki rows.
5. Tests in `kaikki_list_test.go`: a `vi` row kept; an `en` row rejected and counted;
`Hà Nội` kept as `hà nội`; a word listed twice with different `pos` kept once; a malformed
line names its number; `meta` carries a 64-hex `source_sha256` equal to `sha256sum` of the
fixture file, the right `source_rows`, and no `source_commit`/`sources_*` keys.
6. Run against the real file (`scratchpad/kk-vi-edition.jsonl` from this session, or a fresh
download) and record the counts in phase 3.
## Success Criteria
- [x] `go test ./cmd/build-dictionary/` passes; `merged_list*.go` no longer exist.
- [x] Real file builds >30,000 words (measured 34,813) and logs a POS tally.
- [x] `build-dictionary --help` lists `--kaikki`, `--words`, `--out`, `--max-syllables`,
`--min-words` and nothing else.
- [x] `meta.source_sha256` of a build equals `sha256sum` of the input file.
- [x] The fixture path (`--words ... --min-words 150`) is byte-for-byte unaffected in
behaviour: same 205 words from `testdata/fixture-words.txt`.
## Risk Assessment
**A row exceeds the scanner buffer.** Signal: `bufio.Scanner: token too long` on the real
file. Response: pre-decided — use `bufio.Reader` with no line cap rather than guessing a
buffer size; measure the longest line once and note it in the phase result.
**kaikki changes the row shape.** `word` and `lang_code` are wiktextract's stable core fields
and unlikely to move. Signal: zero accepted words → the 30,000 floor fails the build with a
clear message. Response: read the new shape, adjust the struct; the failure mode is loud by
design.
**The file arrives partially.** `curl -f` catches HTTP errors but not a truncated body, and
a body cut exactly on a line boundary parses cleanly. Signal: a malformed final line (error
names it) or a word count under the 30,000 floor (13.8% headroom on 34,813). Response: the
fetch writes to a `.part` name and renames only on success (review finding, Phase 2), so an
interrupted download is never left for the next `make dict`; the floor covers the rest.
plans/260908-1653-kaikki-viwiktionary-corpus/phase-02-fetch-the-newest-file.md
-105
View File
@@ -1,105 +0,0 @@
---
phase: 2
title: "Phase 2: Fetch the newest file"
status: completed
priority: P1
effort: "2h"
dependencies: [1]
---
# Phase 2: Fetch the newest file
## Overview
Point the Makefile and the Docker dictionary stage at kaikki's rolling URL with no checksum,
keep the two build files in agreement, and make sure a failed download cannot pass as a
success anywhere.
## Requirements
- Functional: `make fetch-dict` downloads the kaikki file to `data/kaikki-viwiktionary-vi.jsonl`
with `curl -fL` and fails on any HTTP error; `make dict` builds from it with `--kaikki`.
- Functional: `verify-dict` and `DICT_SHA256` are removed from the Makefile and Dockerfile;
nothing in the repository claims a checksum for this file.
- Functional: the Docker `dict` stage downloads the same URL and runs the same command;
`FIXTURE_DICT=1` keeps working untouched.
- Functional: `web/tests/dictionary-source.test.js` asserts the Makefile and Dockerfile URLs
agree, that the URL is the kaikki viwiktionary `Tiếng Việt` file, and that the builder's
`kaikkiSourceURL` constant is the same URL. The three checksum assertions go.
- Functional: the CI leak guard greps for `kaikki.*\.jsonl$`; `.gitignore` ignores the new
file name.
- Non-functional: no resume flag (`-C -`) — a rolling file can change between attempts, and a
resumed download would splice two versions together.
## Architecture
```make
DICT_URL := https://kaikki.org/viwiktionary/Ti%E1%BA%BFng%20Vi%E1%BB%87t/kaikki.org-dictionary-Ti%E1%BA%BFngVi%E1%BB%87t.jsonl
DICT_SRC := data/kaikki-viwiktionary-vi.jsonl
DICT_OUT := data/noitu.db
fetch-dict:
@mkdir -p data
curl -fL -o $(DICT_SRC).part $(DICT_URL) && mv $(DICT_SRC).part $(DICT_SRC)
dict: $(DICT_SRC)
cd server && go run ./cmd/build-dictionary --kaikki ../$(DICT_SRC) --out ../$(DICT_OUT)
```
The Dockerfile mirrors it with `ARG DICT_URL` only. The URL must stay percent-encoded in
both files: Make and `sh` would otherwise split on the space in `Tiếng Việt`.
Correctness now rests on three checks instead of a checksum: `curl -f` plus an atomic
`.part` rename (HTTP errors and interrupted downloads), the JSON decoder (shape, and
truncation that lands mid-line), the 30,000 floor (content, including a truncation that
lands on a line boundary).
## Related Code Files
- Modify: `Makefile` — variables, `help` text, `fetch-dict`, remove `verify-dict` and its
`.PHONY` entry, `dict` flags
- Modify: `Dockerfile` — `ARG DICT_URL`, remove `ARG DICT_SHA256` and the `sha256sum` line,
download file name, `--kaikki`; stage comment (no longer "pinned … checked by digest")
- Modify: `web/tests/dictionary-source.test.js` — drop checksum tests, keep URL agreement,
add the kaikki-path and builder-constant assertions
- Modify: `.github/workflows/ci.yml` — leak-guard pattern and comment
- Modify: `.gitignore` — `data/kaikki-viwiktionary-vi.jsonl` replaces the undertheseanlp entry
- Modify: `README.md` — Make targets table (`verify-dict` row gone), the manual `curl` and
`build-dictionary` lines
## Implementation Steps
1. Update the Makefile variables and targets; delete `verify-dict`.
2. Mirror in the Dockerfile; rewrite the stage comment to say the file is fetched fresh and
identified by the SHA-256 the builder records.
3. Rewrite the pin test as described. Run `npx vitest run tests/dictionary-source.test.js`.
4. Update the CI leak guard, `.gitignore` and the README lines.
5. Run `make fetch-dict && make dict` (or the raw commands on Windows) from a clean `data/`.
6. Build the image with and without `FIXTURE_DICT=1`; run the CI file checks against the
real one and confirm no `*.jsonl` is inside.
7. Simulate a bad download once: point `DICT_URL` at a 404 and confirm `curl -f` fails the
target; feed the builder a truncated copy of the file and confirm the build fails.
## Success Criteria
- [x] `make fetch-dict` fetches ~62 MB and `make dict` builds >30,000 words.
- [x] `grep -rn DICT_SHA256 --exclude-dir=plans .` finds nothing; `make verify-dict` is not a
target.
- [x] `web/tests/dictionary-source.test.js` passes and fails if either URL is edited alone.
- [x] Both image variants build; the real one passes the CI file checks; no `.jsonl` inside.
- [x] A 404 URL and a truncated file each fail the build with a readable message.
## Risk Assessment
**kaikki.org is a single volunteer-run host.** An outage breaks `make fetch-dict` and the
release image build until it returns. Signal: `curl: (22)` on fetch. Response: wait, or
build from a previously fetched local copy (`make dict` needs only the file); if outages
recur, keep a copy as a release asset and point `DICT_URL` at it — the open question in
`plan.md`.
**Two builds ship different words.** By design. Signal: none needed. Response: `meta`
identifies the bytes; the audit note in phase 3 records what this first build contained.
**The space in the URL.** Signal: `curl` fetching `https://kaikki.org/viwiktionary/Ti%E1%BA%BFng`
and failing. Response: the URL stays percent-encoded in every file, and the pin test checks
that the string contains `Ti%E1%BA%BFng%20Vi%E1%BB%87t`.
plans/260908-1653-kaikki-viwiktionary-corpus/phase-03-measure-the-corpus.md
-81
View File
@@ -1,81 +0,0 @@
---
phase: 3
title: "Phase 3: Measure the corpus"
status: completed
priority: P1
effort: "1h"
dependencies: [1, 2]
---
# Phase 3: Measure the corpus
## Overview
Record what the first kaikki build contains against today's database: the graph numbers, the
words gained and lost, the POS mix, and bot-versus-bot behaviour. Measurement, not a gate —
no rule removes words any more, so there is nothing to audit by hand.
## Requirements
- Functional: a note in this plan directory with the numbers below and the commands that
produced them, plus the `source_sha256` of the file measured so the note is tied to bytes.
- Functional: a diff against the current `data/noitu.db` (copied aside before Phase 2
overwrites it): shared, gained, lost, with 25 random samples of each.
- Functional: playability on both databases: words, syllables, openers, openers with ≥2
continuations, dead ends, and the `internal/bot` real-corpus test output.
- Non-functional: every number reproducible from a written command.
## Architecture
Pre-measured this session on kaikki's 2026-09-06 file (sha256 `51ddc2fbd73cd7e2…`); the
phase re-measures on whatever the build fetched and explains any delta:
| | today | kaikki-vi (2026-09-06) |
|---|---|---|
| words | 26,845 | 34,813 |
| syllables | 5,709 | 6,081 |
| openers ≥2 continuations | 2,787 | 3,163 |
| dead-end syllables | 1,551 | 1,594 |
| shared / gained / lost | | 25,392 / 9,421 / 1,453 |
POS of the 41,507 distinct words: noun 16,219 · verb 9,355 · adj 6,698 · name 5,180 ·
unknown 4,652 · adv 1,024 · phrase 392 · proverb 322 · character 198. `name` and `unknown`
are kept; the log prints the tally so a future decision has the numbers.
## Related Code Files
- Create: `plans/260908-1653-kaikki-viwiktionary-corpus/measurement.md`
- Read only: the pre-switch `data/noitu.db` copy, the new build, `server/internal/bot`
real-corpus tests
## Implementation Steps
1. Before Phase 2 overwrites it, copy today's `data/noitu.db` to a scratch path.
2. After the first `make dict`, record `meta.source_sha256`, `source_rows`,
`source_fetched_at` and the build log's reject and POS lines.
3. Compute the diff and the five graph numbers for both databases (sqlite, one script; keep
the script text in the note).
4. Run `go test ./internal/bot/ -run RealCorpus -v` against both databases (swap the file
under `data/noitu.db`, restore afterwards) and record the game-length and win-rate lines.
5. Look at the 1,453 lost words: they are words in the 2018 scrape that Wiktionary has since
deleted or renamed. Sample 25, note what they look like (expected: misspellings, moved
pages, deleted junk). No action unless the sample is mostly real vocabulary.
## Success Criteria
- [x] `measurement.md` exists with the tables, samples, bot lines, commands and the source
SHA-256.
- [x] Words > 30,000; syllables, openers and ≥2-continuation counts all above today's.
- [x] Bot real-corpus tests pass on the new database; easy-vs-easy game length is not
shorter than today's 12.9 moves.
- [x] The lost-word sample is characterised in one paragraph.
## Risk Assessment
**The fetched file differs from the one measured today.** Certain over time. Signal: counts
off from the table above. Response: record the new numbers and the SHA-256; the deltas are
the point of the note, not a failure.
**Dead ends rise slightly (1,551 → 1,594).** More words bring more rare final syllables.
Signal: already known. Response: the ratio of dead ends to syllables falls (27.2% → 26.2%),
so the graph is denser, not sparser; record it and move on.
plans/260908-1653-kaikki-viwiktionary-corpus/phase-04-relicense-to-4-0.md
-94
View File
@@ -1,94 +0,0 @@
---
phase: 4
title: "Phase 4: Relicense to 4.0"
status: completed
priority: P1
effort: "1h"
dependencies: [3]
---
# Phase 4: Relicense to 4.0
## Overview
Current Wiktionary text is CC BY-SA 4.0, so the derived database is too. Restore the 4.0
legal code, rewrite the attribution chain for a source with no commit to cite, and update
every place that says 3.0 — in the same commit as the first kaikki-built database.
## Requirements
- Functional: `data/LICENSE` is the CC BY-SA 4.0 legal code, byte-identical to the file this
repository shipped before commit `39ef457` (`git show e19b083:data/LICENSE`).
- Functional: `NOTICE` section 2 says 4.0 and names Wiktionary tiếng Việt as the original
work and kaikki.org/wiktextract as the extraction; section 1 byte-identical.
- Functional: `data/ATTRIBUTION.md` records the source URL, that the file is refreshed
weekly and unpinned, that the exact bytes of any build are identified by
`meta.source_sha256`, the three attribution links, and the modifications list.
- Functional: the in-game footer links to the 4.0 deed and says 4.0; the e2e assertion
matches.
- Non-functional: no overclaiming about dates — the attribution names the dump date only as
"the Wikimedia dump kaikki extracted at fetch time; see `source_fetched_at`", not a fixed
date that goes stale.
## Architecture
Attribution chain, all three named:
```
Wiktionary tiếng Việt contributors authors, CC BY-SA 4.0
→ wiktextract / kaikki.org (Tatu Ylonen) extraction; cite LREC 2022 as kaikki requests
→ this project filter + index, data/noitu.db, CC BY-SA 4.0
```
Modifications list (renumbered): 1 language selection (`lang_code = vi`), 2 length filter,
3 content filter, 4 normalization, 5 spelling aliases, 6 added columns and tables,
7 deduplication, 8 dropped fields (senses, translations, POS, categories — everything but
the word form).
## Related Code Files
- Modify: `data/LICENSE` — restore 4.0 text from git history
- Modify: `NOTICE` — section 2 heading, source lines, share-alike sentence
- Modify: `data/ATTRIBUTION.md` — full rewrite of Source, Modifications 1 and 8, reproduce
block (no checksum step)
- Modify: `README.md` — licence table row, dictionary-source paragraph
- Modify: `Dockerfile`, `docs/deployment.md`, `.github/workflows/ci.yml` — 3.0 → 4.0 in comments
- Modify: `server/cmd/build-dictionary/main.go` doc comment, `server/cmd/noitu-server/main.go`
and `server/internal/dictionary/store.go` comments, `store_test.go` seed strings
- Modify: `web/src/lib/components/AttributionFooter.svelte` (deed URL, comment),
`web/src/lib/i18n/vi.js` (`attributionLicense`), `web/e2e/bot-game.spec.js` (regex)
- Verify unchanged: `.github/workflows/ci.yml` required-files check
## Implementation Steps
1. `git show e19b083:data/LICENSE > data/LICENSE`; confirm the first lines read
"Attribution-ShareAlike 4.0 International".
2. Rewrite `NOTICE` section 2 source lines; leave the share-alike paragraph's meaning, change
the version.
3. Rewrite `data/ATTRIBUTION.md` as described. State plainly: "The file is not pinned. Each
build fetches kaikki's current export; the SHA-256 and row count of the file a given
database was built from are recorded in its `meta` table."
4. `grep -rn "3\.0\|by-sa/3" --exclude-dir=plans .` and fix every hit; then the same for
`undertheseanlp`, `DICT_SHA256`, `--merged`, `--sources`.
5. Regenerate `data/noitu.db`; confirm `meta.source_license` says 4.0 and matches the files.
6. Run the full verification set from the previous plan: Go vet/test, web check/test, both
image builds with the CI file checks.
## Success Criteria
- [x] `data/LICENSE` is the 4.0 text; `git diff e19b083 -- data/LICENSE` is empty.
- [x] No file outside `plans/` says CC BY-SA 3.0 or names undertheseanlp.
- [x] `NOTICE` section 1 has no diff.
- [x] `ATTRIBUTION.md` names all three links and states the unpinned-fetch policy.
- [x] `meta.source_license` in the built database says 4.0.
- [x] All verification suites green; the commit contains both the licence files and the
builder change so no intermediate tree mixes 3.0 text with 4.0 data.
## Risk Assessment
**A stale 3.0 or undertheseanlp claim survives.** Signal: the grep in step 4 finds one after
the phase is called done. Response: grep is a step for that reason; run it last.
**The footer wording.** `giấy phép CC BY-SA 3.0` → `4.0` is one string, but the e2e regex
and the deed URL must change with it. Signal: e2e footer test fails in CI. Response: all
three edits are listed above; do them together.
plans/260908-1653-kaikki-viwiktionary-corpus/plan.md
-142
View File
@@ -1,142 +0,0 @@
---
title: "Kaikki viwiktionary corpus"
description: "Replace the 2018 undertheseanlp wiktionary rows with kaikki.org's current extraction of Wiktionary tiếng Việt, fetched unpinned at build time, and move the data licence to CC BY-SA 4.0"
status: completed
priority: P1
effort: "~1d"
tags: [dictionary, data, licensing, build]
created: 2026-09-08
blockedBy: []
blocks: []
---
# Kaikki viwiktionary corpus
## Overview
`data/noitu.db` is built from the `wiktionary` rows of `undertheseanlp/dictionary`: a
2018-12-10 scrape of vi.wiktionary.org, 26,845 words after filtering, pinned by commit and
SHA-256 (plan `260908-1525-dictionary-corpus-switch`, completed today). It is the same
website eight years staler than it needs to be.
This plan switches the source to **kaikki.org's extraction of Wiktionary tiếng Việt** — the
`Tiếng Việt` language file of the `viwiktionary` edition, produced by wiktextract from the
monthly Wikimedia dump (currently 2026-09-01) and refreshed roughly weekly. Same authors,
same website, current text. Measured through the real `build-dictionary` filter this
session: **34,813 words**, a near-superset of today's corpus (25,392 shared, 9,421 gained,
1,453 lost) and 96% of the raw dump (36,200).
Two owner decisions shape the plan:
- **viwiktionary only.** kaikki's English-Wiktionary Vietnamese file (21,244 words, marked
deprecated by kaikki) is not used. The game is Vietnamese; one edition, one attribution.
- **No pin — fetch the newest file every build.** kaikki publishes rolling files with no
archived snapshots, so a checksum pin would break weekly. The owner chose freshness over
reproducibility. The build therefore has to *fail loudly* on a bad download (HTTP error,
malformed JSON, too few words) and *record what it actually got* (SHA-256 and row count in
`meta`) so any shipped database can still be traced to exact bytes.
Consequence: the current text of Wiktionary is CC BY-SA **4.0** (Wikimedia moved from 3.0
on 2023-06-29), so `data/noitu.db` returns to 4.0. The 3.0 chosen this afternoon only ever
applied to the 2018 snapshot.
Evidence: `plans/reports/research-260908-1529-viwiktionary-dump-measured.md` (dump and
kaikki-en measurements) plus this session's measurement of the kaikki-vi file, recorded in
[phase 3](./phase-03-measure-the-corpus.md).
## What changes, in numbers
| | today | after |
|---|---|---|
| words | 26,845 | **34,813** (30,231 if `name` POS were dropped — it is not) |
| syllables | 5,709 | 6,081 |
| syllables with ≥2 continuations | 2,787 | 3,163 |
| dead-end syllables | 1,551 | 1,594 |
| source | 4.8 MB JSONL, 2018, pinned | **62 MB JSONL, current, unpinned** |
| data licence | CC BY-SA 3.0 | **CC BY-SA 4.0** |
| `--min-words` floor | 20,000 | **30,000** |
| build reproducible | yes | **no** — traceable via `meta.source_sha256` |
## The source
```
URL https://kaikki.org/viwiktionary/Tiếng Việt/kaikki.org-dictionary-TiếngViệt.jsonl
(percent-encoded: /viwiktionary/Ti%E1%BA%BFng%20Vi%E1%BB%87t/kaikki.org-dictionary-Ti%E1%BA%BFngVi%E1%BB%87t.jsonl)
Size 62,288,004 bytes on 2026-09-06 — 44,564 rows, 41,507 distinct words, all lang_code "vi"
Row {"word": "trở thành", "pos": "verb", "lang_code": "vi", "lang": "Tiếng Việt", "senses": [...], ...}
```
Only `word` and `lang_code` are read. `pos` is counted for the build log but never filters:
the owner's earlier decision that capitalization does not remove words carries over to the
`name` POS (5,138 name-only words). 5,475 words carry uppercase and 229 lowercase forms
occur twice; `accept()` lowercases and dedupes as it always has.
## Decisions taken
- **One corpus input mode.** `--merged`/`--sources` (undertheseanlp) is replaced by
`--kaikki`, not kept beside it. Two readers for one database is dead weight, as the SQLite
path was. The fixture path `--words` stays.
- **The floor moves to 30,000.** 34,813 measured; a truncated or reshaped download must
fail, and 30,000 is the highest round number with a comfortable margin.
- **Provenance is measured, not asserted.** The reader hashes the file as it streams it and
writes `source_sha256`, `source_rows` and `source_fetched_at` into `meta`. There is no
commit to record, so this is what "which bytes" means from now on.
- **Attribution names three links.** Wiktionary tiếng Việt contributors (authors, CC BY-SA
4.0), wiktextract/kaikki.org by Tatu Ylonen (extraction; kaikki asks for the LREC 2022
citation), and this project. No GitHub intermediary any more.
- **`verify-dict` goes.** There is nothing to verify against. The
Makefile/Dockerfile agreement test keeps guarding the URL and drops the checksum clauses.
## Phases
| # | Phase | Status |
|---|-------|--------|
| 1 | [Read the kaikki file](./phase-01-start.md) | Pending |
| 2 | [Fetch the newest file](./phase-02-fetch-the-newest-file.md) | Pending |
| 3 | [Measure the corpus](./phase-03-measure-the-corpus.md) | Pending |
| 4 | [Relicense to 4.0](./phase-04-relicense-to-4-0.md) | Pending |
Phase 4 lands in the same commit as the first kaikki-built `noitu.db`: a 4.0-derived
database in a tree whose `NOTICE` says 3.0 is the one ordering mistake with a legal
consequence. Phase 3 is measurement, not a gate — there is no drop rule left to audit.
## Success criteria
- [x] `make fetch-dict && make dict` downloads kaikki's current file and builds
`data/noitu.db` with **more than 30,000 words**; a truncated or non-JSON download fails
the build with a message naming the cause.
- [x] `meta` records `source_url`, `source_sha256`, `source_rows`, `source_fetched_at` and
`source_license = CC BY-SA 4.0`; no `source_commit`, `sources_kept` or
`sources_excluded` remain.
- [x] `data/LICENSE` is the CC BY-SA 4.0 legal code; `NOTICE`, `data/ATTRIBUTION.md`, README,
Dockerfile, deployment doc, the builder's licence string and the in-game footer all say
4.0 and credit Wiktionary tiếng Việt contributors and wiktextract/kaikki.org.
- [x] Playability measured and recorded against today's database, including bot game
lengths; nothing regresses below today's numbers.
- [x] `go vet ./... && go test ./...`, `npm run check && npm test`, the Docker image with the
CI file checks and `FIXTURE_DICT=1` are green. e2e: not runnable on this machine (Playwright Chromium download fails); the one changed assertion was updated by hand, CI verifies.
- [x] No file outside `plans/` names undertheseanlp, `--merged`, `--sources` or `DICT_SHA256`.
## Review
Code review (2026-09-08) returned DONE_WITH_CONCERNS; every finding was applied: `fetch-dict`
downloads to a `.part` name and renames on success so an interrupted fetch never feeds the
next `make dict`; a failed `Stat` is an error rather than a zero `source_fetched_at`;
`builder_version` bumped to 3 for the changed meta contract; `.gitignore` covers
`data/*.jsonl`; `ATTRIBUTION.md` states kaikki's redistribution terms; bare JSON literals
are malformed rather than "foreign-language"; the non-EOF read-error line number is exact;
the CI leak guard matches any `.jsonl`; byte-exact reader tests cover a missing final
newline, an HTML error page, a mid-line cut and a bare literal. The reviewer's residual
note stands: with no pin, the build trusts kaikki.org over TLS, and the recorded SHA-256 is
the compensating control; blast radius is game content only.
## Open questions
- kaikki's weekly refresh means two builds a week apart can differ. Accepted by the owner;
`source_sha256` in `meta` is the answer to "which words did this image ship". Should a
release ever need to be rebuilt bit-for-bit, the fallback is to keep the fetched file as a
release asset — a Makefile variable change, not a redesign.
- The `unknown` POS bucket (4,616 words such as `tình báo`, `trọc phú`, `hội quán`) is real
vocabulary the vi extractor could not label. Kept, like everything else.
<!-- slug: kaikki-viwiktionary-corpus -->
plans/260908-2056-wiktionary-dump-corpus-and-meanings/measurement.md
-251
View File
@@ -1,251 +0,0 @@
---
title: "Measurement: first dump-built database against the kaikki one"
date: 2026-09-08
plan: 260908-2056-wiktionary-dump-corpus-and-meanings
status: complete
---
# Measurement: the dump-built database
Both databases were built on 2026-09-08 on the same machine with the same `accept()` filter.
The kaikki one was built with the last kaikki-era builder (`builder_version` 4) from the
export kaikki served that afternoon; the dump one with this plan's builder (`builder_version`
5) from the `latest/` dump, which was the 2026-09-01 run.
## The file measured
| | |
|---|---|
| URL | `https://dumps.wikimedia.org/viwiktionary/latest/viwiktionary-latest-pages-articles.xml.bz2` |
| Resolved to | the 2026-09-01 run (`source_fetched_at` 2026-09-01T11:52:49Z, the file's own mtime kept by `curl -R`) |
| Size | 63,513,513 bytes |
| SHA-256 | `ed66c932f535b0b362d1141c02273b02ed01c4101e856378bd15e88d277bb8d9` (matches the research report's hash of the dated file) |
| `source_pages` | 43,011 |
| Decompressed | 505,982,744 bytes |
## Build log
```
pages 391543, in the main namespace 349461, redirects skipped 3237, without a Vietnamese section 303213
Vietnamese sections: legacy {{-vie-}} 35884, new == {{langname|vi}} == 7127, pages with both 0, titles merged 131
parts of speech: noun 15890, verb 8983, adj 6285, place 3258, pr-noun 1666, n 987, adv 689, phrase 589, idiom 541, v 503, adjc 499, proverb 333, proper noun 159, interj 131, …
headings without a label: cụm động từ 2, giải thích 1, định nghĩa 1
codes that ended a legacy section, commonest: tyz 256, eng 152, mtq 147, nut 49, vi-m 47, nuo 39, tyj 29, kpm 16, mlc 14, tou 14, aav-qal 13, vie-m 12, jra 10, sqi 10, bdq 9
definitions kept 40937 (cut at 200 characters: 515), dropped as empty after stripping 271
templates dropped whole, commonest: alternative spelling of 77, vi-alternative spelling of 40, senseid 37, vie alternative spelling of 36, misspelling of 34, syn of 32, mention 18, en 13, zh 13, synonym of 11
rejected 4: contains a digit
rejected 155: contains punctuation
rejected 6359: fewer than 2 syllables
rejected 162: no Vietnamese letters
rejected 303213: not a Vietnamese-language entry
accepted 36200 distinct words, 35062 with a meaning, (43011 pages, sha256 ed66c932f535b0b362d1141c02273b02ed01c4101e856378bd15e88d277bb8d9) in 32s
generated 1736 spelling aliases (558 skipped as ambiguous or already real words)
wrote data/noitu.db
```
**Wall time: 21–32 s** for the whole build on a laptop, Go's pure-Go bzip2 included. The
two-minute risk did not materialise; the `bzip2 -dc` fallback was not needed.
The counters reconcile: 349,461 main-namespace pages − 3,237 redirects − 303,213 without a
Vietnamese section = 43,011 = `source_pages`. The definition counters are taken after
`accept()`, so they describe words that land: 40,937 definitions kept against 40,842 rows,
the difference being senses past the cap on the 131 merged title pairs. The "codes that
ended a legacy section" line lists only language codes (tyz, eng, mtq, nut, …), which is the
check that no heading code is missing from the maps and cutting sections short.
The legacy count (35,884) is one page short of the research report's 35,885 because one page
opens its Vietnamese section after a level-2 heading of another dialect and is read as that
one; the new-dialect count differs by one for the same reason (7,127 vs 7,128). "Pages with
both" is zero, so the merge path for mixed pages stays untested against real data.
## Graph metrics, old vs new
| metric | kaikki (v4) | dump (v5) | change |
|---|---|---|---|
| words | 34,813 | **36,200** | +1,387 |
| syllables | 6,081 | 6,172 | +91 |
| syllables that open a word | 4,487 | 4,600 | +113 |
| …with ≥ 2 continuations | 3,163 | 3,246 | +83 |
| dead-end syllables | 1,594 | 1,572 | −22 |
| aliases | 1,664 | 1,736 | +72 |
| database size on disk | 2.2 MB | 6.9 MB | meanings text |
No metric regresses.
## Corpus diff
| | |
|---|---|
| shared | **34,813** — every kaikki word is in the dump build |
| gained | 1,387 |
| lost | **0** |
The first build lost 15 words (`tây tạng`, `thượng hải`, `nam kinh`, …). All were pages that
run `{{-vie-}}{{-pron-}}{{vie-pron|…}}{{-place-}}` together on one line, which the scanner
read as no section at all. Fixed in phase 1 (headings are read from a line that starts with
`{{-`, several per line); the second build lost none.
Gained, 25 at random: nghiến ngấu, hồn ai nấy giữ, pháo đùng, trèo cao té đau, ngồi rồi, tối
như hũ nút, tiêu tức, kinh kỳ, rón rón, tự phê bình, vàng hương, trơ mắt ếch, thìa khóa, qua
cầu rút ván, tinh khí, rộng chân rộng cẳng, say lử cò bợ, miệng ăn, sảo thai, phỉ dạ, vắng
như chùa bà banh, cọc tìm trâu, óng a óng ánh, lồm lộp, phơi phóng. Reduplicatives and
idioms, as the research predicted.
## Bot vs bot, 60 games each
Run with `go test ./internal/bot/ -run RealCorpus -v -count=1` against each database copied
to `data/noitu.db`. (The kaikki database needed an empty `meanings` table and a
`meaning_count` row added to a scratch copy: the v5 store refuses a v4 file, by design.)
| | kaikki | dump |
|---|---|---|
| hard beats easy | 100% (3.4 moves) | 95% (3.5 moves) |
| medium beats easy | 88% | 82% |
| hard beats medium | 60% (2.9 moves) | 72% (4.1 moves) |
| easy vs easy, game length | 16.8 moves | 13.7 moves |
| hard decision latency | n=74 p95 2.6 ms max 3.4 ms | n=68 p95 2.5 ms max 4.0 ms |
Both pass the ladder test's thresholds. The differences are within what a 60-game sample
moves between runs; the corpus is 4% larger and the graph slightly better connected.
## Meanings
| | |
|---|---|
| `meaning_count` (senses) | 40,842 |
| `words_with_meaning` | 35,062 of 36,200 = **96.9%** (floor in `verify`: 60%) |
| senses per word | 1: 30,454 · 2: 3,773 · 3: 583 · 4: 167 · 5 (the cap): 85 |
| definitions cut at 200 characters | 587 |
| definitions dropped as empty after stripping | 942 |
| senses with an empty part-of-speech label | 4,849 = **11.9%** |
Labels, by sense: danh từ 14,471 · động từ 7,804 · tính từ 5,839 · (empty) 4,849 · địa danh
3,702 · danh từ riêng 1,859 · phó từ 758 · cụm từ 544 · thành ngữ 446 · tục ngữ 329 · thán từ
112 · đại từ 47 · liên từ 44 · số từ 17 · giới từ 11 · tiền tố 7 · trợ từ 3.
The empty share is above the plan's 10% target, but the unmapped-heading list has been worked
through: every code seen more than once is mapped (see below). What remains empty is the
research report's "no-pos" class — pages whose Vietnamese section has definitions under no
part-of-speech heading at all (`sun phát`, `kiến vàng`, `long đong` in the sample). Nothing
in the wikitext gives those a label.
**Headings without a label, by frequency:** cụm động từ 2, classifier 1, det 1, giải thích 1,
prop 1, reading 1, tiếng anh 1, tyj 1, định nghĩa 1, đồng nghĩa khác âm 1. Nothing worth a
map entry.
**Templates dropped whole, ten commonest:** `vi-han form of` 222, `vi-nom form of` 163,
`alternative spelling of` 96, `senseid` 79, `rfdef` 63, `abbreviation of` 48,
`vi-alternative spelling of` 45, `vie alternative spelling of` 44, `vie-han form of` 41,
`misspelling of` 36. The "form of" and "spelling of" family is the one worth teaching next:
a definition that is only `{{vi-alt sp|hoá thạch}}` strips to nothing today, and `hóa thạch`
is one of the 1,138 words without a meaning for that reason. `rfdef` is a request for a
definition and correctly yields none.
### Deviations from the plan's label map, decided in phase 1 from the frequency tables
- `pron` on this wiki is **pronunciation** (36,533 legacy headings, 5,501 new), not pronoun.
The pronoun code is `pronoun` (158 + 141), also `per-pronoun`. The plan's `pron → đại từ`
would have labelled 36,000 pronunciation sections as pronouns.
- The new dialect uses `{{section|code}}` (3,506 headings) as well as `{{ĐM|code}}`, and the
shorthand codes `n` (647 + 9,488 wiki-wide) and `v` (353 + 3,774) beside `noun`/`verb`.
- `{{-dfn-}}` (4,615) is a "definitions" heading placed *under* a part-of-speech heading, so
it is transparent to the label rather than a heading of its own.
- Bare headword lines `{{vi-noun}}`, `{{vie-noun}}`, `{{vi-pr-noun}}` set the label when
their code is a part of speech; `{{vi-pron}}`, `{{vi-etym-sino}}` and kin do not.
- Two definition-shaped templates beyond the plan's list were taught because they top the
definition-line template counts: `{{nhãn|…}}`/`{{context|…}}`/`{{term|…}}` as label
spellings, `{{n-g|…}}` as a non-gloss definition, and `{{see-entry|x}}`/`{{like-entry|x}}`
(13,122 + 1,038 definition lines wiki-wide) rendered as `Xem x`.
- Definition lines without a space after `#` (`#{{term|cái cân}} Đồ dùng…`) are accepted;
only `##`, `#:`, `#*`, `#;` are sub-items.
## The 50-sense sample
Drawn with `SELECT word, pos, gloss FROM meanings ORDER BY RANDOM() LIMIT 50` from the
first build and read by hand. Verdicts: **fine 43, terse 5, wrong 2.**
| # | word | pos | gloss | verdict |
|---|---|---|---|---|
| 1 | quý tộc | danh từ | Họ dòng sang. | terse |
| 2 | khuyến mại | động từ | Xúc tiến việc mua bán hàng hóa, cung ứng dịch vụ bằng cách dành cho khách hàng những lợi ích nhất định. | fine |
| 3 | bất đắc dĩ | tính từ | Không có sự lựa chọn khác. | fine |
| 4 | hắc điếm | danh từ | (từ cổ) Quán trọ, khách sạn, nơi tạm trú (có thể do kẻ xấu lập ra nhằm cướp của, giết người khi có dịp). | fine |
| 5 | đèn vách | danh từ | Đèn dầu hoả treo trên vách nhà. | fine |
| 6 | chuẩn cơm mẹ nấu | cụm từ | Món hợp khẩu vị, ăn mãi không ngán. | fine |
| 7 | tác ác | động từ | Làm việc ác. | fine |
| 8 | ỉ ê | tính từ | Từ gợi tả tiếng khóc nhỏ, dai dẳng và ỉ eo một cách khó chịu (thường nói về trẻ con) | fine |
| 9 | dơ bẩn | tính từ | (Phương ngữ) Xem nhơ bẩn | fine (cross-reference) |
| 10 | tốt bộ | | Chỉ đẹp có bề ngoài. | fine, no label |
| 11 | hãm hại | động từ | Làm hại, giết chết bằng thủ đoạn ám muội. | fine |
| 12 | thuần hậu | tính từ | Chất phác hiền hậu. | fine |
| 13 | phúc bạc | | Phúc mỏng, ít phúc (không phải bạc là trắng, dù tác giả có ý đối với chữ má đào). | fine, no label |
| 14 | chặt chẽ | tính từ | Có quan hệ khăng khít, gắn kết với nhau. | fine |
| 15 | long đong | | Vất vả, nay đây mai đó, hay gặp nhiều rủi ro. | fine, no label |
| 16 | hồng bì | danh từ | Thứ cây cùng họ với cam, quít, quả nhỏ, da vàng, có lông nhung, vị chua ngọt. | fine |
| 17 | thế chiến i | danh từ riêng | Xem Chiến tranh thế giới thứ nhất. | fine (cross-reference) |
| 18 | khuyên nhủ | động từ | Khuyên bảo ân cần. | fine |
| 19 | thường thới hậu a | địa danh | Một xã thuộc huyện Hồng Ngự, tỉnh Đồng Tháp, Việt Nam. | fine |
| 20 | gạo sen | danh từ | Hạt trắng hình hạt gạo, ở đầu nhị đực của hoa sen, dùng để ướp chè. | fine |
| 21 | kèn trống | danh từ | Kèn và trống thường sử dụng trong đám ma. | fine |
| 22 | lộng quyền | động từ | Làm việc vượt quá quyền hạn của mình, lấn cả quyền hạn của người cấp trên. | fine |
| 23 | quay đĩa | danh từ | (Khẩu ngữ) máy quay đĩa (nói tắt) | fine |
| 24 | tam giáo | danh từ | (Id.). Ba thứ đạo ở Trung Quốc thời trước. | terse (a source abbreviation survived) |
| 25 | điều dưỡng viên | danh từ | Người phụ trách công tác điều dưỡng, … cho đến phục hồi, trị… | fine (cut at 200) |
| 26 | bán tống bán tháo | cụm từ | (Khẩu ngữ) Xem bán đổ bán tháo | fine (cross-reference) |
| 27 | kiểu cách | danh từ | Kiểu mẫu và cách thức. | fine |
| 28 | hiểm ác | tính từ | Ác một cách ngấm ngầm. | fine |
| 29 | ca nô | danh từ | Thuyền máy cỡ nhỏ, mạn cao, có buồng máy, buồng lái, dùng chạy trên quãng đường ngắn. | fine |
| 30 | bút pháp | danh từ | (cũ) Phong cách viết chữ Hán. | fine |
| 31 | thuần việt | tính từ | Có nguồn gốc từ người Việt. | fine |
| 32 | sun phát | | Muối của a-xít sun-phu-rích. | fine, no label |
| 33 | kiến vàng | | Loài kiến nhỏ, màu vàng, đốt đau. | fine, no label |
| 34 | trinh phú | địa danh | Một xã thuộc huyện Kế Sách, tỉnh Sóc Trăng, Việt Nam. | fine |
| 35 | lạnh gáy | tính từ | Xem lạnh người. | fine (cross-reference) |
| 36 | giải phóng | động từ | Làm cho được tự do, cho thoát khỏi địa vị nô lệ hoặc tình trạng bị áp bức, kiềm chế, ràng buộc. | fine |
| 37 | đẹp duyên | động từ | (kiểu cách) Xem kết duyên | fine (cross-reference) |
| 38 | thanh đạm | tính từ | (Ẩm thực) Giản dị, không có những món cầu kì hoặc đắt tiền. | fine |
| 39 | suy tàn | động từ | Ở trạng thái suy yếu và tàn lụi, không còn sức sống. | fine |
| 40 | tràng thạch | danh từ | (Địa lý học). | **wrong** — the definition after the label was a template the stripper dropped |
| 41 | toàn bộ | danh từ | Tất cả các phần, các bộ phận của một chỉnh thể. | fine |
| 42 | bát tiên | danh từ riêng | Tám vị tiên là Hán Chung Ly, … hay được vẽ trên màn trướng. | fine |
| 43 | ba mũi giáp công | danh từ | Tiến công bằng ba hình thức kết hợp: quân sự, chính trị và binh vận. | fine |
| 44 | sở ước | | Điều mình mong được. | fine, no label |
| 45 | giao diện lập trình ứng dụng | danh từ | (programming) Đặc tả quy định cách … | terse (English label from a pasted template) |
| 46 | ngộ gió | | Bị cảm vì gặp gió. | fine, no label |
| 47 | ảo tung chảo | tính từ | Cảm giác khó tin. | terse |
| 48 | nhóm bếp | | Đốt lửa cho củi bắt đầu cháy trong bếp. | fine, no label |
| 49 | trần hy tăng | danh từ riêng | Xem Trần Bích San | fine (cross-reference) |
| 50 | cung ứng | động từ | Cung cấp đáp ứng nhu cầu, thường là của sản xuất, hoặc của hành khách. | fine |
Two wrong out of fifty, for different reasons; well under the one-in-five threshold that
would have sent the stripper back to phase 1. The one systematic gap is a label followed by
a template-only definition (#40), which leaves a bare `(label).` — a follow-up for the
stripper, template by template.
## Words without a meaning
1,138 words (3.1%). Three read at random:
- `hóa thạch` — one definition, `# {{vi-alt sp|hoá thạch}}`, an alternative-spelling
template the stripper does not know.
- `lóng lánh`, `khoái trá` — the page has only pronunciation, paronyms and a `{{-see-}}`
section with `{{like-entry|…}}` as a bullet, not a `#` line. No definition to carry.
## Verification run alongside this measurement
- `go vet ./... && go test ./... -race`: green.
- `npm run check && npm test`: green (183 tests).
- `npm run test:e2e` and the two Docker image variants: see the plan's session log.
## After code review
The review (`plans/reports/code-reviewer-260908-2210-wiktionary-dump-corpus-and-meanings.md`)
found no critical issue; its findings were fixed in the same working tree: the bot-game e2e
assertion now reads the drawn opening's sense from the fixture list instead of assuming
`danh từ`; the chain-row styling is scoped to the outer list so the sense list renders as a
numbered list; the stripper drops format characters (bidi overrides, zero-width spaces) as
well as controls; a pre-v5 database is refused with a message that says to rebuild; a lone
short template parameter is no longer mistaken for a language code; the definition counters
are taken after `accept()`; section-ending codes are tallied; the meanings lookup happens once
per move; `aria-controls` is set only while the panel exists. The numbers above are from the
build after those fixes.
plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-01-start.md
-192
View File
@@ -1,192 +0,0 @@
---
phase: 1
title: "Phase 1: Read the dump"
status: completed
priority: P1
effort: "6h"
dependencies: []
---
# Phase 1: Read the dump
## Overview
Replace the kaikki JSONL reader in `build-dictionary` with a streaming reader for the
Wikimedia `pages-articles.xml.bz2` dump that yields, per Vietnamese-section page, the title
for `accept()` and the stripped definition lines for the meanings table.
## Requirements
- Functional: `--dump <file>` reads a bzip2-compressed MediaWiki XML export. Pages with
`ns != 0` or a `<redirect>` element are skipped and counted. Every other page's `<title>`
and latest `<revision><text>` are handed to the section scanner.
- Functional: the scanner finds the Vietnamese section in either dialect — `{{-vie-}}` or a
level-2 heading whose text is `{{langname|vi}}` — and stops at the section's end: for
legacy pages, the next `{{-xxx-}}` template whose code is not in the section-code set; for
new pages, the next level-2 heading. Pages with no Vietnamese section are counted, not
read. Pages with a section in *both* dialects are read once, first dialect wins, counted.
- Functional: the title goes to `accept()` unchanged from today. Part of speech labels seen in
the section (`{{-noun-}}`, `{{ĐM|noun}}`, `{{vi-noun}}` and kin, `pr-noun`/`place`
included) are tallied for the log and never filter.
- Functional: within the section, every line starting with `# ` (exactly one `#`, then a
space — not `#:`, `#*`, `##`) is a definition. It is stripped (see Architecture), capped at
200 runes with an ellipsis, and appended to the page's senses up to 5. Empty results are
dropped. Senses are kept in page order.
- Functional: each sense carries the part of speech of the heading it sits under, as a
Vietnamese label from a fixed map keyed by the heading code — the same codes in both
dialects (`{{-noun-}}`, `{{ĐM|noun}}`, `{{vi-noun}}` → `danh từ`). Starting map: `noun` danh
từ, `verb` động từ, `adj` tính từ, `adv` phó từ, `pr-noun` danh từ riêng, `place` địa danh,
`pron` đại từ, `num` số từ, `conj` liên từ, `prep` giới từ, `intj` thán từ, `part` trợ từ,
`phrase` cụm từ, `idiom` thành ngữ, `prov` tục ngữ, `abbr` viết tắt, `prefix` tiền tố,
`suffix` hậu tố, `char` chữ. A definition under a heading outside the map, or before any
heading, has an empty label and is kept; unmapped codes are counted for the log. Step 1's
frequency list decides what else the map needs. (Validation session 1.)
- Functional: two pages whose titles normalize to the same word (`Việt Nam` and `việt nam`
both being real pages) merge: the word is one entry, senses concatenate in page order and
the cap applies to the union.
- Functional: provenance is measured during the one streaming pass: SHA-256 of the compressed
bytes as read, `source_pages` (pages with a Vietnamese section, redirects excluded),
`source_fetched_at` from the file's modification time. `source_rows` is gone.
- Functional: the build log reports pages seen, ns0 pages, redirects skipped, pages with a
Vietnamese section per dialect, POS tally, definitions kept, definitions dropped as empty
after stripping, and the ten commonest template names dropped whole.
- Functional: `--kaikki` and `kaikki_list.go` and its tests are removed. Exactly one of
`--dump` / `--words` must be given, same rule as today.
- Non-functional: a stream that is not bzip2, XML that ends mid-page, or a `<text>` element
the decoder cannot read is an error naming the page title or byte offset. A count of
Vietnamese-section pages under 20,000 is an error naming the count, distinct from the word
floor, so a parser that silently misses a dialect is caught before `--min-words` is.
- Non-functional: memory is one page at a time. The XML decoder is used token by token; only
`title`, `ns`, `redirect` and `text` of the current page are held.
- Non-functional: wall time under two minutes on the real dump on a laptop. Measure and
record; see Risk.
## Architecture
```
--dump <file.xml.bz2>
│ os.Open → io.TeeReader(f, sha256) → bufio.Reader → compress/bzip2 → xml.NewDecoder
▼
for each <page>:
ns==0, no <redirect> ─no─► count, skip
section := vietnameseSection(text) // dialect-aware slice of the wikitext
section == "" ─yes─► count "no Vietnamese section", skip
word, syllables, reason, ok := accept(title)
senses := definitions(section) // []sense{pos, gloss}: stripped, ≤5, ≤200 runes each
pos tally from section headings
if ok: words[word] = entry{…}; meanings[word] = merge(meanings[word], senses)
```
`type sense struct { pos, gloss string }` lives in `wikitext.go`; `pos` is the mapped
Vietnamese label or empty.
Files:
```
server/cmd/build-dictionary/
dump.go readDump: bz2+xml streaming, page loop, provenance, dumpSourceURL const
wikitext.go vietnameseSection, definitions, stripWikitext, posLabels, posLabelMap
dump_test.go byte-exact fixtures: one legacy page, one new-dialect page, a redirect,
a foreign-only page, a truncated stream, a non-bzip2 file
wikitext_test.go table tests for the stripper and the section boundaries
```
The section-code set for legacy pages is the list of `{{-xxx-}}` codes that are *headings*
rather than language switches: `etym, pron, noun, verb, adj, adv, pr-noun, place, phrase,
idiom, prov, syn, synonym, ant, trans, ref, reference, see, der, info, num, pron, conj,
prep, intj, part, abbr, char, hanzi, nom, …`. Build it from the real dump in this phase:
extract every `{{-xxx-}}` code, list them by frequency, and classify by hand; the language
codes are 2–3 letters and the section codes are English abbreviations, so the split is
readable. Record the set in `wikitext.go` with the frequency it was seen at.
The stripper, in order:
1. Drop `<!-- … -->`, `<ref …>…</ref>`, `<ref … />`, any other tag pair or lone tag.
2. Templates by a depth counter over `{{`/`}}`. For each outermost template, split on `|`
outside nested braces and brackets, take the name:
- `label`, `lb`, `gloss`, `qualifier`, `q` → `(` join of positional params after a leading
`vi` `)`;
- `l`, `vi-l`, `w`, `m`, `link` → the last positional param (display text if any);
- `place` → positional params after `vi`, each with a leading `[a-z]/` removed, joined by
`, `;
- anything else → dropped, name counted.
3. Links: `[[Thể loại:…]]`/`[[Category:…]]` dropped; `[[a|b]]` → `b`; `[[a]]` → `a`;
external `[http… label]` → `label`.
4. `'''`, `''` removed. `&nbsp;` and the common entities decoded.
5. Control characters (`unicode.IsControl`) removed; whitespace collapsed; trimmed. A result
that is only punctuation is empty.
6. Longer than 200 runes: cut at the last space before 200 and append `…`.
## Related Code Files
- Create: `server/cmd/build-dictionary/dump.go`, `wikitext.go`, `dump_test.go`,
`wikitext_test.go`
- Delete: `server/cmd/build-dictionary/kaikki_list.go`, `kaikki_list_test.go`
- Modify: `server/cmd/build-dictionary/main.go` — package doc, `config.kaikki` → `dump`,
flag text, `run()` switch, `runFromKaikkiList` → `runFromDump`, `entry` gains nothing (the
meanings map travels beside `words` into `finish()` — phase 3 persists it),
`builderVer = "5"`
- Modify: `server/cmd/build-dictionary/main_test.go` — `fixtureSource` writes a small bzip2
XML dump instead of JSONL (Go has no bzip2 *writer*: commit a tiny `testdata/mini-dump.xml.bz2`
built once with `bzip2`, plus its uncompressed source beside it for review)
- Modify: `server/cmd/build-dictionary/filter.go` — `rejectNotVietnamese` comment now refers
to a page with no Vietnamese section; the reason string can stay
## Implementation Steps
1. Download the current dump once locally (phase 2's URL, `curl -fLR`). Write a throwaway
`go run` that streams it and prints every `{{-xxx-}}` code with counts and every level-2
heading text with counts. Build the section-code set and the dialect markers from that
output; keep the numbers in comments.
2. Write `wikitext.go`: `vietnameseSection`, `posLabels`, `definitions`, `stripWikitext`.
Table-test the stripper on the three samples in `plan.md` plus: nested templates, a
`<ref>` mid-sentence, a link with a category, a definition that is only `{{rfdef|vi}}`
(must be empty), a 300-rune definition (must end in `…` at a word boundary), a legacy
page whose senses sit under `{{-noun-}}` then `{{-verb-}}` (labels `danh từ`, `động từ` in
order), a new-dialect page with `=== {{ĐM|pr-noun}} ===` (label `danh từ riêng`), a
definition under an unmapped heading (empty label, sense kept).
3. Write `dump.go`: the streaming loop, provenance, counters, the error shapes listed under
Non-functional. Test against `testdata/mini-dump.xml.bz2` (six pages, both dialects, a
redirect, an English-only page) and byte-truncated copies of it.
4. Rewire `main.go`; delete the kaikki files; make `go vet ./... && go test ./cmd/...` green.
5. Run against the real dump with `--out /tmp/dump.db`. Read the log: pages per dialect should
be within a few percent of 35,885 / 7,128 for the 2026-09-01 dump; accepted words near
36,200. Record wall time. Keep this database for phase 6.
6. Sample 50 senses at random from the log (or from the phase-3 table) and read them. Fix any
stripper rule that is clearly wrong across many entries; note the rest for phase 6.
## Success Criteria
- [x] `go run ./cmd/build-dictionary --dump ../data/viwiktionary-latest-pages-articles.xml.bz2 --out /tmp/dump.db`
completes; the log shows ~43,000 Vietnamese-section pages, both dialects non-zero,
~36,200 accepted words, and a definitions-kept count above 30,000.
- [x] The stripper table tests pass on all listed shapes; `# {{rfdef|vi}}` yields no sense;
labels come out in Vietnamese for both dialects and empty for an unmapped heading.
- [x] On the real dump, senses with an empty label are under 10% of all senses, or the
unmapped-code list has been worked through.
- [x] A truncated dump, a non-bzip2 file and a mid-page cut each fail with a message naming
the cause; a fixture with 3 Vietnamese pages fails the 20,000-page check by name.
- [x] `grep -rn kaikki server/` finds nothing.
- [x] Wall time on the real dump recorded in the phase-6 measurement file.
## Risk Assessment
**Go's bzip2 is slow.** Signal: step 5 takes longer than two minutes. Response: keep the
reader on an `io.Reader`, add `--dump` acceptance of a plain `.xml` (sniff the two magic
bytes `BZ`), and have the Makefile pipe `bzip2 -dc` in; the Docker `dict` stage on alpine has
`bzip2`. Decide in this phase, not later.
**The section-code set is incomplete.** A legacy heading code missing from the set is read as
a language switch and truncates the Vietnamese section early: definitions after it are lost,
the word is not. Signal: the definitions-kept count is well below the page count, or the
sample in step 6 shows senses cut off. Response: the step-1 frequency list is the source of
truth; every code seen more than ~20 times must be classified.
**Definitions in a template the stripper does not know.** The whole sense is dropped. Signal:
"definitions dropped as empty" is large, or a common template name tops the dropped list.
Response: teach the stripper that template if it is a definition-shaped one (`place`,
`label` are the known cases); otherwise accept the loss and record it in phase 6.
**Both dialects on one page.** Rare; first dialect wins and the page is counted. Signal: the
counter is not small. Response: read both sections and merge senses — a small change to
`vietnameseSection` returning a slice.
plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-02-fetch-the-dump.md
-104
View File
@@ -1,104 +0,0 @@
---
phase: 2
title: "Phase 2: Fetch the dump"
status: completed
priority: P1
effort: "2h"
dependencies: [1]
---
# Phase 2: Fetch the dump
## Overview
Point the Makefile, the Docker `dict` stage, the pin test, the CI leak guard and the README's
raw commands at Wikimedia's rolling `latest` dump instead of kaikki's JSONL, keeping the
three URL copies in agreement and a failed download unable to pass as a success.
## Requirements
- Functional: `DICT_URL` is
`https://dumps.wikimedia.org/viwiktionary/latest/viwiktionary-latest-pages-articles.xml.bz2`;
`DICT_SRC` is `data/viwiktionary-latest-pages-articles.xml.bz2`. `fetch-dict` uses
`curl -fLR` — `-R` keeps the server's modification time, which is the dump's generation
time and becomes `source_fetched_at` — to a `.part` name renamed on success, as today.
- Functional: `dict` runs `build-dictionary --dump ../$(DICT_SRC)`.
- Functional: the Dockerfile `dict` stage mirrors the URL and the flag; `FIXTURE_DICT=1` is
untouched. `curl` gains `-R` there too.
- Functional: `web/tests/dictionary-source.test.js` asserts the Makefile and Dockerfile URLs
agree, that the URL matches
`^https://dumps\.wikimedia\.org/viwiktionary/latest/viwiktionary-latest-pages-articles\.xml\.bz2$`,
that the builder's `dumpSourceURL` constant in `dump.go` is the same string, and that README
and `data/ATTRIBUTION.md` quote it.
- Functional: the CI leak guard rejects any `\.bz2$` or `\.xml$` inside the image; the
comment says the dump, not the export. `.gitignore` covers `data/*.bz2`, `data/*.bz2.part`,
`data/*.xml` (for the phase-1 fallback) and drops the `.jsonl` lines.
- Functional: README's "Without make" block, Make targets table and Setup section describe
a ~61 MB monthly dump; the `fetch-dict` help line says the same.
- Non-functional: still no resume flag. `latest` can be repointed between two attempts.
## Architecture
```make
DICT_URL := https://dumps.wikimedia.org/viwiktionary/latest/viwiktionary-latest-pages-articles.xml.bz2
DICT_SRC := data/viwiktionary-latest-pages-articles.xml.bz2
fetch-dict:
@mkdir -p data
curl -fLR -o $(DICT_SRC).part $(DICT_URL) && mv $(DICT_SRC).part $(DICT_SRC)
dict: $(DICT_SRC)
cd server && go run ./cmd/build-dictionary --dump ../$(DICT_SRC) --out ../$(DICT_OUT)
```
Correctness rests on: `curl -f` plus the `.part` rename (HTTP errors, interrupted
downloads); the bzip2 decoder (a truncated stream errors at EOF); the XML decoder (a cut
mid-page); the 20,000-page check and the 30,000-word floor (content).
## Related Code Files
- Modify: `Makefile` — header comment, `DICT_URL`, `DICT_SRC`, `help`, `fetch-dict`, `dict`
- Modify: `Dockerfile` — `dict` stage comment, `ARG DICT_URL`, `curl -fsSLR`, file name,
`--dump`
- Modify: `web/tests/dictionary-source.test.js` — URL pattern, builder file and constant name
- Modify: `.github/workflows/ci.yml` — leak guard pattern and comment; the `image` job's
comment about "the real upstream release" still holds
- Modify: `.gitignore`
- Modify: `README.md` — Setup, Make targets, Without make, Architecture table's dictionary
row wording if it names the export
## Implementation Steps
1. Update the Makefile; run `make fetch-dict` on a clean `data/` and check the file's mtime is
the dump's, not now.
2. Mirror in the Dockerfile.
3. Rewrite the pin test; run `npx vitest run tests/dictionary-source.test.js`. Edit one URL
copy alone and confirm it fails.
4. Update the CI leak guard, `.gitignore`, README.
5. `make dict`; then `docker build .` and `docker build --build-arg FIXTURE_DICT=1 .`; run
the CI file checks against the real image and confirm no `.bz2`/`.xml` inside.
6. Point `DICT_URL` at a 404 once and confirm `fetch-dict` fails; feed `dict` a copy of the
dump cut at 40 MB and confirm the build fails naming the cause.
## Success Criteria
- [x] `make fetch-dict` downloads ~61 MB with the server's mtime; `make dict` builds >30,000
words with meanings.
- [x] `web/tests/dictionary-source.test.js` passes and fails when any one copy is edited.
- [x] Both image variants build; the real one passes the CI file checks; nothing matching
`\.bz2$|\.xml$` inside.
- [x] `grep -rniI kaikki --exclude-dir=plans .` finds nothing outside `plans/`.
- [x] A 404 URL and a truncated file each fail the build readably.
## Risk Assessment
**`latest/` mid-repoint.** Wikimedia updates the `latest` symlinks file by file as a run
completes. Fetching during the run can return the previous month's file, which is correct,
or in principle a 404 for a moment. Signal: `curl: (22)`. Response: retry; nothing to fix.
**Dump size grows.** Irrelevant to correctness; the README's "~61 MB" goes stale slowly.
Signal: none. Response: update the number when it is off by a quarter.
**The Docker build now decompresses bzip2 in Go inside alpine.** Same code, same speed as
locally. If phase 1 chose the `bzip2 -dc` fallback, the stage needs `apk add bzip2` and the
pipe; the leak guard's `.xml` clause is why an uncompressed intermediate is also checked.
plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-03-meanings-in-the-database.md
-114
View File
@@ -1,114 +0,0 @@
---
phase: 3
title: "Phase 3: Meanings in the database"
status: completed
priority: P1
effort: "4h"
dependencies: [1]
---
# Phase 3: Meanings in the database
## Overview
Persist the senses phase 1 extracts in a `meanings` table, load them in `dictionary.Store`
behind a `Meanings(word)` method, and let the fixture word list carry meanings so tests and
the e2e suite have some.
## Requirements
- Functional: schema gains
```sql
CREATE TABLE meanings (
word TEXT NOT NULL,
ord INTEGER NOT NULL,
pos TEXT NOT NULL, -- Vietnamese part-of-speech label, '' when the heading was unmapped
gloss TEXT NOT NULL,
PRIMARY KEY (word, ord)
) WITHOUT ROWID;
```
`ord` is the 0-based sense order from the page. No foreign key pragma; `verify()` checks
the join instead, as it does for aliases. (Validation session 1: `pos` column.)
- Functional: `finish()` takes the meanings map beside `words`; `writeTo` inserts them in the
same transaction. `meta` gains `meaning_count` (rows) and `words_with_meaning`.
- Functional: `verify()` adds: no meaning row whose word is missing from `words`; no empty
gloss; no gloss over 200 runes; `words_with_meaning` is at least 60% of `word_count` for a
`--dump` build (fixture builds are exempt — the check is passed a flag, or keyed on the
source table prefix).
- Functional: `dictionary.Store` loads `meanings` into `map[string][]Sense` ordered by
`ord`, where `type Sense struct { Pos, Gloss string }` is exported from the `dictionary`
package, and exposes `Meanings(word string) []Sense` returning nil for a word with none.
`Resolve` first: the caller passes a canonical word, as it does for `FirstSyllable`.
Startup log gains the meanings count next to `words`.
- Functional: `--words` accepts an optional meaning column: `word<TAB>sense<TAB>sense…`,
where a sense is `pos|gloss` or just `gloss` (the pipe never survives stripping, so it is a
safe separator). Lines without a tab have no meaning. `testdata/fixture-words.txt` gains hand-written
meanings for the words the e2e suite plays (phase 5 says which) and for at least half the
list, so the fixture database exercises the same store paths as the real one.
- Non-functional: `Store` memory grows by the text size, a few MB. Startup time unchanged
in practice (one more ordered scan).
- Non-functional: `builder_version = "5"` (set in phase 1; this is the contract it names).
## Architecture
```
build-dictionary dictionary.Store
words map[string]entry ─┐ words map[string]wordInfo
meanings map[string][]sense ─┼─► sqlite ─► meanings map[string][]Sense
aliases map[string]string ─┘ Meanings(word) []Sense
```
The store's `validate()` already cross-checks `word_count`; add the same for
`meaning_count` so a database whose meanings table was truncated on disk is refused at
startup rather than served silently without meanings.
## Related Code Files
- Modify: `server/cmd/build-dictionary/main.go` — `finish`, `write`, `writeTo`, `verify`,
`runFromWordList` (tab-separated senses), `sourceSpec` unchanged
- Modify: `server/cmd/build-dictionary/main_test.go` — meanings written and verified; a
fixture list with tabs; a `verify` failure on an orphan meaning row
- Modify: `server/internal/dictionary/store.go` — `meanings` field, `loadMeanings`,
`Meanings()`, `validate` count check, `MeaningCount()`
- Modify: `server/internal/dictionary/store_test.go` — hand-built databases gain the table;
`Meanings` returns ordered senses and nil; a mismatched `meaning_count` is refused
- Modify: `server/cmd/noitu-server/main.go` — startup log line
- Modify: `testdata/fixture-words.txt` — meanings column; header comment documents the format
- Modify: `web/e2e/fixture-dictionary.js` — it reads the list one word per line
(`fixture-dictionary.js:15-19`); cut each line at the first tab before trimming, or the
e2e graph gains words that are really `word<TAB>sense` strings (validation session 1)
## Implementation Steps
1. Schema, insert, meta and `verify` in the builder; tests first for the orphan-row and
empty-gloss failures.
2. `runFromWordList` tab parsing; a test that a line with two tabs yields two senses in order,
`danh từ|…` splits into label and gloss, a cell without a pipe has an empty label, and a
line without a tab yields none. Update `fixture-dictionary.js` in the same step.
3. Store: load, method, validate; tests.
4. Fixture list: add meanings. Rebuild `data/fixture.db` (`make fixture-dict`) and start the
server against it; the log shows the count.
5. Rebuild the real database from phase 1's dump and confirm `verify` passes the 60% rule.
## Success Criteria
- [x] `go test ./cmd/build-dictionary ./internal/dictionary -race` green.
- [x] `sqlite3 data/noitu.db 'SELECT COUNT(*) FROM meanings'` is above 40,000 and
`words_with_meaning` above 60% of words (phase 6 records the actual).
- [x] `store.Meanings("học sinh")` on the real database returns a non-empty ordered slice
whose first sense has `Pos == "danh từ"`; on a word with no `#` line it returns nil.
- [x] `npm run test:e2e` still builds its word graph from the fixture list correctly (no
tab-carrying "words").
- [x] The fixture database has meanings for the e2e words and the server starts on it.
## Risk Assessment
**The 60% rule is a guess.** Research counted 5,764 of 43,013 pages with no POS marker, and
some of those have no `#` line either; the true coverage is unknown until phase 1 runs.
Signal: `verify` fails on the real dump with a coverage in the 50s. Response: measure, set
the floor a comfortable margin under the measurement, and record why in the code comment.
The rule exists to catch a stripper that suddenly returns nothing, not to demand quality.
**Fixture meanings drift from fixture words.** A word renamed in the list loses its meaning
silently. Signal: an e2e assertion on a meaning fails. Response: the builder's fixture path
logs words without meaning; keep the list short and hand-checked.
plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-04-meanings-on-the-wire.md
-108
View File
@@ -1,108 +0,0 @@
---
phase: 4
title: "Phase 4: Meanings on the wire"
status: completed
priority: P1
effort: "3h"
dependencies: [3]
---
# Phase 4: Meanings on the wire
## Overview
Carry a word's senses in the two messages that introduce a word to the client —
`PlayedWord` and `GameStarted` — regenerate the Go and JS types and the binary fixtures,
and have the room attach the senses where it already renders those messages.
## Requirements
- Functional: `proto/noitu/v1/game.proto` gains
```proto
// One definition of a word as Wiktionary gives it: the part of speech it
// sits under, in Vietnamese ("danh từ"), empty when the heading was not one
// the builder knows, and the definition stripped of markup at at most 200
// characters. Plain text both: the client renders them as text, never as
// markup.
message Sense {
string pos = 1;
string gloss = 2;
}
message PlayedWord {
…
// At most five. Empty when the dictionary has no definition for the word.
repeated Sense meanings = 7;
}
message GameStarted {
…
repeated Sense opening_meanings = 9;
}
```
(Validation session 1: `Sense` instead of a bare string, so the label is a field.)
- Functional: `wsapi.Dictionary` gains `Meanings(word string) []dictionary.Sense`.
`game.Dictionary` does not change; the engine never reads a meaning. `convert.go` gains
`Senses([]dictionary.Sense) []*noituv1.Sense`.
- Functional: `convert.PlayedWord` takes the senses (`PlayedWord(m game.Move, byMe bool,
meanings []dictionary.Sense)`) and its one caller, `sendTurnUpdate` (`room.go:903`, also
used by the resume path), passes `r.dict.Meanings(move.Word)`; `sendGameStarted` fills
`OpeningMeanings` from `r.dict.Meanings(r.opening)`. Both paths are per recipient already;
the lookup happens once per move, before the per-seat loop.
- Functional: the reconnect path (`GameStarted` + last `TurnUpdate`) carries meanings
without further change; a test asserts a resumed client sees the opening's and last word's
senses.
- Functional: `buf generate` regenerates `server/gen/` and `web/src/lib/proto/`;
`go test ./internal/wsapi -update` regenerates `proto/testdata/`; the JS decode test reads
the new field.
- Functional: the hand-built dictionaries in `wsapi` tests implement `Meanings`; at least
one returns senses so the assertion is not vacuous.
- Non-functional: frame size grows by up to ~1 KB per move. Nothing in the client or server
caps frames near that.
## Architecture
```
room.handleSubmit ── engine.Submit ──► Move ──► meanings := r.dict.Meanings(move.Word)
for each seat: PlayedWord(move, byMe, meanings)
room.sendGameStarted ─────────────────────────► OpeningMeanings: r.dict.Meanings(r.opening)
```
The bot room path renders through the same `sendTurnUpdate`, so bot moves carry meanings
without a second change.
## Related Code Files
- Modify: `proto/noitu/v1/game.proto`
- Regenerate: `server/gen/noitu/v1/game.pb.go`, `web/src/lib/proto/**`, `proto/testdata/*`
- Modify: `server/internal/wsapi/room.go` — `Dictionary` interface, `sendGameStarted`,
the `PlayedWord(` call in `sendTurnUpdate`
- Modify: `server/internal/wsapi/convert.go` — `PlayedWord` signature, `Senses`
- Modify: `server/internal/wsapi/*_test.go` — fake dictionaries gain `Meanings`; new
assertions on `Played.Meanings` and `OpeningMeanings`, including after a resume
- Modify: `web/tests/game-wire.test.js` — the fixture decode asserts the new fields
## Implementation Steps
1. Edit the schema; `buf lint`; `buf generate`.
2. Interface and room changes; fix compile errors in tests by adding `Meanings` to fakes.
3. Add the assertions; `go test ./internal/wsapi -update` then `go test ./... -race`.
4. `cd web && npm test` — the wire test decodes the regenerated fixtures.
5. `make proto-check` — committed generated code matches the schema.
## Success Criteria
- [x] `buf lint` clean; `make proto-check` clean.
- [x] A bot game and a room game deliver `Played.Meanings` with `Pos` and `Gloss` for a word
the fake dictionary has senses for, and an empty list for one it does not.
- [x] A resumed session's `GameStarted.OpeningMeanings` and replayed `TurnUpdate.Played.Meanings`
are populated.
- [x] `proto/testdata/` fixtures regenerated and the JS suite decodes them.
## Risk Assessment
**Field numbers.** `PlayedWord` uses 1–6, `GameStarted` 1–8; 7 and 9 are free and nothing
is reserved in either. Signal: `buf breaking` complaint. Response: none expected; adding a
repeated field is wire-compatible, and there is one client.
**Meanings on `PlayerEliminated.suggestions`.** Out of scope by the plan's non-goals; a
reviewer may ask. Response: the suggestions are shown to somebody who just lost the turn,
a list of words, and a meaning under each would be a second UI. Not this plan.
plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-05-meanings-in-the-client.md
-133
View File
@@ -1,133 +0,0 @@
---
phase: 5
title: "Phase 5: Meanings in the client"
status: completed
priority: P1
effort: "5h"
dependencies: [4]
---
# Phase 5: Meanings in the client
## Overview
Show a word's meanings under it in the chain, open for the newest word, closed for the rest,
with a click on any word toggling its own; keep the open/closed set in the store so the
rules are testable without a browser, and cover the behaviour end to end.
## Requirements
- Functional: `ChainEntry` gains `meanings: {pos: string, gloss: string}[]`, filled from
`played.meanings` and `value.openingMeanings`.
- Functional: the store keeps `expanded: Set<string>` of words whose meaning is open, and
`toggleMeaning(word)`. Any number of words may be open at once. Rules (validation
session 1: they apply to every word, with or without definitions):
- `gameStarted`: `expanded = {openingWord}`.
- `turnUpdate` with a word: remove the previous newest word (`chain[chain.length-1].word`
before the push), add the new word. A `turnUpdate` without a word (an elimination)
changes nothing.
- `toggleMeaning(word)`: flip membership.
- `reset()` clears the set. A resumed session gets `gameStarted` then one `turnUpdate` and
ends up with only the last word open, which is the same as a fresh one.
- Functional: `ChainHistory.svelte` renders every word as a `<button type="button">` with
`aria-expanded` and `aria-controls` pointing at its panel. The panel is shown when the word
is in `expanded`: an `<ol class="meanings">` with one `<li>` per sense rendered as
`(pos) gloss`, or `gloss` alone when `pos` is empty; when the word has no senses the panel
is a single `<p class="meanings none">` with `t.meaningNone`. Everything else in the row —
who played it, badges, points, the correction note — is unchanged.
- Functional: strings in `web/src/lib/i18n/vi.js`: `meaningShow` / `meaningHide` for the
button's `aria-label` (`Xem nghĩa của {word}` / `Ẩn nghĩa của {word}`) and `meaningNone`
(`Chưa có nghĩa`).
- Functional: the auto-scroll effect keeps working — the newest row is at the top and its
open list is what the scroll lands on.
- Functional: `history-export.js` is unchanged (non-goal).
- Non-functional: the meanings are rendered as text (`{sense}`), never with `{@html}`.
- Non-functional: the row stays legible on a phone: the list wraps under the word at full
row width (`flex-basis: 100%`), muted colour, ~0.85rem, numbered by the `<ol>`.
- Non-functional: `prefers-reduced-motion` respected if any open/close transition is added;
the simplest is none.
## Architecture
```
game.svelte.js
state.chain[i].meanings from the wire
state.expanded Set<string>, client-only
toggleMeaning(word)
ChainHistory.svelte
<li>
<button class="word" type="button" aria-expanded={open} aria-controls={id}
aria-label={fill(open ? t.meaningHide : t.meaningShow, { word: entry.word })}
onclick={() => game.toggleMeaning(entry.word)}>{entry.word}</button>
… by / meta / corrected as today …
{#if open}
{#if entry.meanings.length}
<ol class="meanings" {id}>
{#each entry.meanings as sense}<li>{sense.pos ? `(${sense.pos}) ` : ''}{sense.gloss}</li>{/each}
</ol>
{:else}
<p class="meanings none" {id}>{t.meaningNone}</p>
{/if}
{/if}
</li>
```
`id` is derived from the row's index in the chain, not from the word, so two rows never
share one (a word is never played twice, but the opening word could in principle be typed
as a later variant).
The e2e helper `chainWords` selects `ol li .word`; the class stays on the button so the
helper and every existing spec keep working, and the nested `<ol class="meanings">` has no
`.word` inside it.
## Related Code Files
- Modify: `web/src/lib/stores/game.svelte.js` — `ChainEntry` typedef, `expanded`,
`toggleMeaning`, the `gameStarted`/`turnUpdate`/`reset` branches
- Modify: `web/src/lib/components/ChainHistory.svelte` — markup and styles
- Modify: `web/src/lib/i18n/vi.js` — three strings
- Modify: `web/tests/game-store.test.js` — the expansion rules above, each as a case
- Modify: `web/tests/i18n.test.js` — only if it enumerates keys
- Modify: `web/e2e/helpers.js` — `openMeanings(page)` returning the words whose list is
visible; `web/e2e/bot-game.spec.js` — newest open, previous closed after a move, click
toggles; `web/e2e/pvp-game.spec.js` — the other browser sees the same word open
- Modify: `testdata/fixture-words.txt` — meanings for the words the specs play (phase 3)
## Implementation Steps
1. Store first, with tests: opening open; new word closes previous and opens itself; a word
with no meanings is opened the same way; elimination leaves the set alone; toggle flips
and two words can be open together; reset clears; the resume sequence ends with one open.
2. Component markup and styles; check both themes and a 360px viewport.
3. i18n strings; run the copy test.
4. e2e: fixture meanings, helper, three assertions. CI runs the browser; locally, run what
Playwright allows.
5. `npm run check && npm test`.
## Success Criteria
- [x] `npm run check && npm test` green; store tests cover all seven rules.
- [x] In a bot game the opening word's meaning is open; after the bot's first move only the
bot's word is open; clicking the opening word opens it again while the bot's stays
open, and clicking the bot's word closes it.
- [x] Senses render as `(danh từ) …`; a sense with an empty label renders the gloss alone; a
word without meanings opens to `Chưa có nghĩa`.
- [x] e2e specs assert the three behaviours and pass in CI.
- [x] Nothing in the transcript export changed.
## Risk Assessment
**The button steals focus from the word input.** The README says focus is left alone while
a player types, and a chain row that grabs focus on render would break that. Signal: typing
interrupted after a move. Response: never call `focus()` in the component; the button is
only focusable by the user's own click or Tab.
**Long meanings push the chain off screen on a phone.** Five senses of 200 characters is a
tall row. Signal: the newest row fills the viewport. Response: the cap is server-side and
already chosen; if it proves too tall, `max-height` with `overflow-y: auto` on the list is
a style change. Do not shorten text client-side — it would differ from what the server sent.
**A resumed session opens two words.** `gameStarted` opens the opening word; the replayed
`turnUpdate` must close it. Covered by the rule ordering and a store test using the resume
sequence.
plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-06-measure-and-attribute.md
-122
View File
@@ -1,122 +0,0 @@
---
phase: 6
title: "Phase 6: Measure and attribute"
status: completed
priority: P1
effort: "3h"
dependencies: [2, 3]
---
# Phase 6: Measure and attribute
## Overview
Record what the first dump-built database contains against the kaikki one, sample the
stripper's output, and rewrite every attribution surface so it names the Wikimedia dump,
drops kaikki.org and wiktextract, and stops claiming the database holds only word forms.
## Requirements
- Functional: `measurement.md` beside this plan, same shape as the kaikki plan's: the file
measured (URL, SHA-256, `source_pages`, `source_fetched_at`, size), the build log, the
graph metrics (words, syllables, openers, openers ≥2, dead ends) old vs new, bot-vs-bot at
60 games, the corpus diff (shared / gained / lost with samples), wall time of the build,
and the meanings numbers: `meaning_count`, `words_with_meaning` and its percentage, the
distribution of senses per word, the share of senses with an empty part-of-speech label
and the unmapped heading codes by frequency, the ten commonest dropped template names, and
a 50-sense random sample read by a person with each marked fine / terse / wrong.
- Functional: `data/ATTRIBUTION.md` — source table names the dump
(`viwiktionary-latest-pages-articles.xml.bz2`, monthly, unpinned), removes the wiktextract
row and the LREC citation, states the chain is now Wiktionary tiếng Việt contributors →
this project. Modifications list: item 1 becomes "section selection — the Vietnamese
section of each page, either markup dialect; redirects skipped"; a new item describes the
definition text: taken from `#` lines, markup stripped, templates other than the listed
ones removed, capped at five senses of 200 characters, each labelled with the Vietnamese
name of the part-of-speech heading it sat under, so the text is an *excerpt and
modification* of the entry, not the entry; item 8 ("dropped fields") says everything else
— examples, translations, pronunciations, etymologies, categories — is dropped, and the
sentence "contains only word forms, not meanings" is deleted. The
reproduce section names the new commands.
- Functional: `NOTICE` — affected artifacts name the dump file; the extraction lines go;
"identified by the SHA-256 recorded in the database's meta table" stays.
- Functional: README — licence section drops wiktextract/kaikki; Setup and the raw commands
already changed in phase 2; a sentence under the game description says the chain shows
each word's Wiktionary meaning.
- Functional: `docs/deployment.md` "Updating the dictionary" says dumps.wikimedia.org and
monthly.
- Functional: the in-game footer (`AttributionFooter.svelte`, `attributionSource`) already
says Wiktionary tiếng Việt and links vi.wiktionary.org; verify and leave.
- Functional: `verify` in the builder and the pin test are the only two places allowed to
encode numbers from this measurement (the coverage floor; the URL). Nothing else.
- Non-functional: the measurement is a record, not a gate — the plan's decision that nothing
regresses is checked against the numbers, and a regression is reported to the owner, not
silently accepted or silently fixed.
## Architecture
```sh
cp data/noitu.db /tmp/kaikki.db # before the first dump build
make fetch-dict && make dict
cd server && go test ./internal/bot/ -run RealCorpus -v -count=1 # per database
```
```sql
SELECT COUNT(*) FROM words; SELECT COUNT(*) FROM syllables;
SELECT COUNT(*) FROM (SELECT first FROM words GROUP BY first HAVING COUNT(*) >= 2);
SELECT COUNT(*) FROM syllables WHERE out_degree = 0;
SELECT COUNT(*) FROM meanings; SELECT COUNT(DISTINCT word) FROM meanings;
SELECT pos, COUNT(*) FROM meanings GROUP BY pos ORDER BY 2 DESC; -- '' is the unmapped share
SELECT n, COUNT(*) FROM (SELECT word, COUNT(*) n FROM meanings GROUP BY word) GROUP BY n;
SELECT word, gloss FROM meanings ORDER BY RANDOM() LIMIT 50;
ATTACH '/tmp/kaikki.db' AS old;
SELECT COUNT(*) FROM words w JOIN old.words o ON o.word = w.word; -- shared
SELECT word FROM words WHERE word NOT IN (SELECT word FROM old.words) ORDER BY RANDOM() LIMIT 25; -- gained
SELECT word FROM old.words WHERE word NOT IN (SELECT word FROM words) ORDER BY RANDOM() LIMIT 25; -- lost
```
## Related Code Files
- Create: `plans/260908-2056-wiktionary-dump-corpus-and-meanings/measurement.md`
- Modify: `data/ATTRIBUTION.md`, `NOTICE`, `README.md`, `docs/deployment.md`
- Verify only: `web/src/lib/components/AttributionFooter.svelte`, `web/src/lib/i18n/vi.js`
## Implementation Steps
1. Before the first real dump build, copy today's `data/noitu.db` aside.
2. Build; run the queries and the real-corpus bot tests on both databases; fill
`measurement.md`.
3. Read the 50 senses; mark each; if more than ~10 are wrong for the same reason, that is a
phase-1 stripper fix, made now, and the sample re-drawn.
4. Rewrite the four documents. Run `npx vitest run tests/dictionary-source.test.js` (README
and ATTRIBUTION quote the URL).
5. `grep -rniI "kaikki\|wiktextract\|jsonl\|only word forms" --exclude-dir=plans .` finds
nothing.
6. Plan status via `ak plan`; journal.
## Success Criteria
- [x] `measurement.md` complete with every table above; words ≥ 34,813 and no graph metric
below the kaikki database's, or the regression is written up and shown to the owner.
- [x] Meaning coverage recorded; the 50-sense sample recorded with verdicts.
- [x] `data/ATTRIBUTION.md`, `NOTICE`, README and `docs/deployment.md` describe the dump and
the definition text accurately; the pin test passes.
- [x] The grep in step 5 is empty.
## Risk Assessment
**The dump has fewer words than kaikki for some syllables.** Research measured the dump as a
near-superset (36,200 vs 34,813 through the same filter), but the two are different months
by the time this runs. Signal: `lost` is in the thousands. Response: list them, look for a
parser cause (a section boundary cut, a dialect miss) before accepting; a genuine wiki
deletion is accepted and recorded.
**The stripper sample is bad.** Signal: more than a fifth of 50 marked wrong. Response: fix
the stripper for the commonest cause in this plan; ship with the rest recorded; the
follow-up is a plan of its own, because it is template-by-template work against a moving
wiki.
**Attribution says less than the law wants.** CC BY-SA 4.0 asks for attribution, a licence
link, and an indication of modifications. The definition text makes the "modifications"
line load-bearing: the database now redistributes edited excerpts of the entries. The
ATTRIBUTION rewrite above is the compliance; a reviewer should read it against the licence
text once. Signal: none in code. Response: do the review.
plans/260908-2056-wiktionary-dump-corpus-and-meanings/plan.md
-283
View File
@@ -1,283 +0,0 @@
---
title: "Wiktionary dump corpus and word meanings"
description: "Build data/noitu.db from the Wikimedia dump of Wiktionary tiếng Việt instead of kaikki.org's export, carry each word's definitions into the database, and show them in the chain: click to toggle, newest word open by default"
status: completed
priority: P1
effort: "~2.5d"
tags: [dictionary, data, proto, web, licensing, build]
created: 2026-09-08
blockedBy: []
blocks: []
---
# Wiktionary dump corpus and word meanings
## Overview
`data/noitu.db` is derived from kaikki.org's wiktextract export of Wiktionary tiếng Việt
(plan `260908-1653-kaikki-viwiktionary-corpus`, completed today): 34,813 words, word forms
only, fetched unpinned. Two things change here.
**The source becomes the Wikimedia dump itself.** `dumps.wikimedia.org/viwiktionary/` is the
file kaikki extracts from. Reading it directly removes the intermediary, its weekly refresh
and its `vi` extractor, which today loses ~1,400 real words the dump has (measured in
`plans/reports/research-260908-1529-viwiktionary-dump-measured.md`: **36,200** accepted
titles against kaikki's 34,813, with reduplicatives such as `rưng rức`, `nháo nhác`, `óc ách`
among the difference). The cost is a wikitext parser of our own — both markup dialects the
wiki currently mixes — instead of a JSON decoder.
**The database gains meanings.** Each word carries its Wiktionary definitions, stripped of
markup, capped at five senses of 200 characters. The client shows them under the word in the
chain: the newest word's meaning is open by default, a new word closes the previous one and
opens its own, and a click on any word toggles its meaning. This is the reason the source
matters: kaikki's glosses are pre-cleaned, but the dump's definitions cover the words kaikki
drops, and a stripper for `# ...` lines is a bounded job.
Owner decisions taken before this plan was written:
- **Track `latest/`, unpinned.** Same posture as today: `curl` whatever
`viwiktionary-latest-pages-articles.xml.bz2` currently is, record its SHA-256 in `meta`.
Dated directories exist and would pin honestly; the owner chose freshness. Wikimedia's dump
is monthly, so "fresh" now means monthly rather than weekly.
- **All senses, capped, each with its part of speech.** Every definition line of the
Vietnamese section travels, cut at 5 senses and 200 characters each, and each sense carries
the Vietnamese label of the heading it sat under (`danh từ`, `động từ`, `tính từ`, `danh từ
riêng` …), shown as a prefix: `(danh từ) Trẻ em học tập ở nhà trường.` A heading the
builder cannot map yields an empty label, never a dropped sense.
- **Every word is clickable.** A word with no definition still opens, to one line saying
`Chưa có nghĩa`, so the chain behaves the same for every word.
- **Nothing is dropped by label.** Carries over from the previous two plans: part of speech
and capitalization are tallied in the build log and never filter. `Danh từ riêng`
(`pr-noun`) labels are counted, not acted on.
- **One corpus input mode.** `--kaikki` is replaced by `--dump`, not kept beside it. `--words`
stays for the fixture and gains an optional meaning column so the e2e suite can see one.
## What changes, in numbers
| | today (kaikki) | after (dump) |
|---|---|---|
| words | 34,813 | **~36,200** (measured on the 2026-09-01 dump; phase 6 records the actual) |
| meanings | none | one to five per word; coverage measured in phase 6 (~5,700 pages have no `#` line and will have none) |
| source | 62 MB JSONL, weekly, unpinned | **~61 MB `.xml.bz2`, monthly, unpinned** |
| parser | `encoding/json`, 3 fields | `compress/bzip2` + `encoding/xml` + a wikitext section/definition scanner |
| `--min-words` floor | 30,000 | 30,000 (unchanged: 36,200 measured, and a dialect the parser misses is ~7,000 pages) |
| data licence | CC BY-SA 4.0 | CC BY-SA 4.0 (the database now carries definition *text*, so the share-alike obligation is more literal, not different) |
| wire | `PlayedWord` 6 fields | `message Sense {pos, gloss}`; `PlayedWord.meanings`, `GameStarted.opening_meanings` |
| `builder_version` | 4 | 5 |
## The source
```
URL https://dumps.wikimedia.org/viwiktionary/latest/viwiktionary-latest-pages-articles.xml.bz2
Dated https://dumps.wikimedia.org/viwiktionary/20260901/viwiktionary-20260901-pages-articles.xml.bz2
63,513,513 bytes, MD5 6c2491e703e7d946f23a405996b4d172 (published), monthly on the 1st
Shape <mediawiki><page><title/><ns/><id/>[<redirect title=""/>]<revision><text>wikitext</text></revision></page>…
```
391,543 pages; 349,461 in namespace 0; **43,013 with a Vietnamese section**; 3,237 of those
are redirects (834 case-only, `mặt trời` → `Mặt Trời`) and are skipped, because the target
page is read on its own and lowercased by `accept()`.
Two markup dialects are live and the mix shifts monthly:
| | legacy (35,885 pages) | new (7,128 pages) |
|---|---|---|
| Vietnamese section opens | `{{-vie-}}` | `== {{langname\|vi}} ==` |
| section closes | next `{{-xxx-}}` whose code is *not* a known section code (`-eng-`, `-fra-`, `-tyz-` …) | next level-2 heading `== … ==` |
| POS heading | `{{-noun-}}`, `{{-verb-}}`, `{{-pr-noun-}}`, `{{-place-}}` … | `=== {{ĐM\|noun}} ===`, `{{vi-noun}}`, `{{vi-pr-noun}}` … |
| definition | a line starting `# ` (not `#:`, `#*`) under a POS heading | same |
Sample definitions as they sit in the dump and as they must come out:
```
{{-noun-}} … # [[chỗ|Chỗ]] [[râm]] [[mát]], do [[trời]] có [[mây]] hoặc do không bị [[nắng]] [[chiếu]].
→ pos "danh từ" gloss "Chỗ râm mát, do trời có mây hoặc do không bị nắng chiếu."
=== {{ĐM|pr-noun}} === … # {{place|vi|thủ đô|c/Việt Nam}}.
→ pos "danh từ riêng" gloss "thủ đô, Việt Nam."
# {{label|vi|thuộc lịch sử}} Một [[tỉnh]] cũ của [[Việt Nam]] vào nửa cuối thế kỷ XIX.
→ pos "danh từ riêng" gloss "(thuộc lịch sử) Một tỉnh cũ của Việt Nam vào nửa cuối thế kỷ XIX."
```
The client renders a sense as `(pos) gloss`, or `gloss` alone when the label is empty.
## Decisions taken
- **Parse in Go, inside `build-dictionary`.** `compress/bzip2` and `encoding/xml` are stdlib;
the Docker `dict` stage and the Makefile keep one command. Go's bzip2 is pure Go and slow
(tens of MB/s); a ~400 MB decompressed stream is well under a minute. If phase 1 measures
more than two minutes, the fallback is `bzip2 -dc` in the Makefile feeding a plain `.xml`
— a flag change, recorded as a risk below, not a redesign.
- **The stripper is small and lossy on purpose.** Links keep their display text; bold and
italic markers go; `<ref>` and comments go; `{{label|vi|x}}`/`{{lb|vi|x}}`/`{{gloss|x}}`
become `(x)`; `{{l|vi|x}}`, `{{vi-l|x}}`, `{{w|x}}` become `x`; `{{place|vi|a|b}}` keeps its
positional parameters after the language code with any `c/` prefix removed; **every other
template is dropped whole**, nesting handled by a depth counter. What survives is trimmed
and whitespace-collapsed; an empty result is not a sense. This will leave some definitions
terse or odd. Phase 6 samples 50 and records what the stripper got wrong; fixing template
by template is a follow-up, not this plan.
- **Meanings live in their own table** — `meanings(word, ord, pos, gloss)` — and are loaded
into memory by the store with the words. The label is a column, not baked into the gloss:
the 200-character cap applies to the definition, and the client decides how a label is
shown. Ten kilobytes of text per hundred words is a few
megabytes for the corpus; a per-move query would be a second code path for nothing.
- **The engine never sees a meaning.** `game.Dictionary` is unchanged. `wsapi.Dictionary`
gains `Meanings(word) []dictionary.Sense` and the room attaches them when it renders a `PlayedWord`
or a `GameStarted`, the same place `by_me` is decided. A reconnect already replays only the
opening and the last move, and both carry meanings, so nothing new is needed there.
- **Expansion state is client-only.** Which words are open is UI state, like the theme. The
store keeps a set of open words — any number may be open at once; `gameStarted` opens the
opening word, a `turnUpdate` with a word closes the previous newest and opens the new one,
a click toggles. These rules apply to every word, with or without a definition. The
transcript export is unchanged.
- **Meaning text is plain text end to end.** It is rendered as text, never as HTML; the
builder strips markup, the client escapes as Svelte does by default. Wiki text is
user-generated content and travels through the same sanitizer path as any other string
the server has not written itself: the builder caps length, drops control characters and
collapses whitespace so nothing the store loads can be shaped like a chat injection.
## Phases
| # | Phase | Status |
|---|-------|--------|
| 1 | [Read the dump](./phase-01-start.md) | Completed |
| 2 | [Fetch the dump](./phase-02-fetch-the-dump.md) | Completed |
| 3 | [Meanings in the database](./phase-03-meanings-in-the-database.md) | Completed |
| 4 | [Meanings on the wire](./phase-04-meanings-on-the-wire.md) | Completed |
| 5 | [Meanings in the client](./phase-05-meanings-in-the-client.md) | Completed |
| 6 | [Measure and attribute](./phase-06-measure-and-attribute.md) | Completed |
Phases 1 and 3 land together: a reader that extracts definitions and a schema that has
nowhere to put them is half a change. Phase 2 can land with them or just after. Phases 4
and 5 are one wire change and its consumer; the schema regenerates once. Phase 6 is
measurement and the attribution rewrite, and the attribution must land in the same commit
as the first dump-built database, because `data/ATTRIBUTION.md` today says the database
"contains only word forms, not meanings", which stops being true.
## Non-goals
- Dropping words by proper-noun label, capitalization or part of speech.
- Keeping kaikki as a fallback or second source.
- Meanings for the suggestions shown to an eliminated player, in the transcript export, or
anywhere outside the chain.
- A dictionary lookup UI. The meaning is shown for words that were played.
- Pinning the dump. Recorded as the open question it already was.
## Success criteria
- [x] `make fetch-dict && make dict` downloads `viwiktionary-latest-pages-articles.xml.bz2`
and builds `data/noitu.db` with more than 30,000 words and a meaning for the large
majority of them; the build log reports pages seen, pages with a Vietnamese section,
pages per dialect, redirects skipped, POS tallies and definition counts.
- [x] A truncated or non-bzip2 download, an XML stream that ends mid-page, and a page count
far below the norm each fail the build with a message naming the cause.
- [x] `meta` records `source_url`, `source_sha256`, `source_pages`, `source_fetched_at`,
`meaning_count` and `builder_version = 5`; `source_rows` is gone.
- [x] `Sense`, `PlayedWord.meanings` and `GameStarted.opening_meanings` are in the schema,
the generated Go and JS trees, and the binary fixtures under `proto/testdata/`.
- [x] In a bot game and a two-browser online game the newest word's meaning is open, the
previous word's closes when a new one arrives, and clicking any word toggles its own.
Senses show as `(danh từ) …`; a word with no definition opens to `Chưa có nghĩa`.
- [x] `data/ATTRIBUTION.md`, `NOTICE`, README, `docs/deployment.md`, the Dockerfile and the
builder's constants name the Wikimedia dump and no longer name kaikki.org or
wiktextract; the "only word forms" claim is replaced by an accurate description of the
definition text carried.
- [x] `go vet ./... && go test ./... -race`, `npm run check && npm test`, `npm run test:e2e`
(CI) and both Docker image variants are green; the CI leak guard rejects a `.bz2` or
`.xml` inside the image.
- [x] Phase 6's measurement is recorded beside this plan: word count, meaning coverage,
graph metrics and bot game lengths against the kaikki database, plus the 50-sense
stripper sample.
## Open questions
- **Unpinned `latest/`.** Wikimedia repoints `latest` once a month; a dump run can also fail
partway and leave `latest` on the previous month, which is harmless. Accepted by the owner,
same as for kaikki. `source_sha256` and `source_fetched_at` (the file's server-side
modification time, kept by `curl -R`) identify a build. Should reproducibility ever be
wanted, the dated URL is a one-variable change.
- **Definitions that are only a template.** `# {{place|vi|thủ đô|c/Việt Nam}}.` strips to
`thủ đô, Việt Nam.` — readable, not prose. Definitions that are *only* an unknown template
strip to nothing and the sense is dropped; if that leaves a word with no sense, the word
has no meaning shown. Phase 6 counts how many and lists the commonest template names so
the next round knows which to teach the stripper.
- **Headings the label map does not know.** `{{-xxx-}}` and `{{ĐM|xxx}}` codes outside the
map give an empty label; the sense is kept. Phase 6 lists unmapped codes by frequency so
the map can grow; a code seen on more than a few hundred definitions is added in phase 1.
- **Dialect drift.** The wiki is migrating legacy `{{-vie-}}` pages to `== {{langname|vi}} ==`.
Both are parsed; the build log prints the count per dialect so a month where one falls to
zero is visible. A third form appearing would show as a drop in `source_pages` and,
eventually, the floor.
## Validation Log
### Session 1 — 2026-09-08
Verification (Full tier, 6 phases; claims checked by reading the code while the plan was
written, then re-grepped): 14 checked, 13 verified, 1 failed, 0 unverified.
- FAILED → fixed: `web/e2e/fixture-dictionary.js` reads `testdata/fixture-words.txt` as one
word per line (`fixture-dictionary.js:15-19`); the tab-separated meaning column must be
stripped there. Phase 3 now lists it as a definite change, not a conditional one.
- Corrected: `PlayedWord(` has one caller, `room.go:903` inside `sendTurnUpdate`; the resume
path reuses it. Phase 4 no longer implies a second call site.
- Verified: `wsapi.Dictionary` at `room.go:268`, `sendGameStarted` at `room.go:744`,
resume replay at `room.go:1190-1195`, `Store.validate` word-count check at `store.go:213`,
`reset()` at `game.svelte.js:198`, `chainWords` selector `ol li .word` at
`e2e/helpers.js:184`, `TestDifficultyLadderRealCorpus` and `TestHardChooseLatencyRealCorpus`
in `bot/realcorpus_test.go`, `web/tests/game-wire.test.js` decodes `proto/testdata`,
`PlayedWord` fields 1–6 and `GameStarted` 1–8 with nothing reserved, `web/tests/i18n.test.js`
does not enumerate keys (phase 5's conditional line resolves to no change), Go 1.25.
Questions asked: 4.
| Question | Decision | Effect |
|---|---|---|
| Open state when another word is opened | Independent toggles; only the automatic close of the previous newest | Phase 5 rules unchanged; stated explicitly |
| Part-of-speech label on senses | **Prefix each sense with its POS** | `meanings.pos` column; `message Sense {pos, gloss}`; heading→label map in phase 1; client renders `(pos) gloss` |
| Meanings in the transcript download | No, chain only | Non-goal stands |
| A word with no definition | **Clickable, opens to `Chưa có nghĩa`** | Every word is a button; `meaningNone` string; auto-open applies to all words |
### Whole-Plan Consistency Sweep
Re-read `plan.md` and all six phase files after propagation. Searched for `no toggle`,
`no button`, `nothing to click`, `[]string`, `repeated string meanings`, `(word, ord, gloss)`,
`if it parses`, `wherever PlayedWord`. All occurrences reconciled to the decisions above. No
unresolved contradictions.
### Session 2 — 2026-09-08, implementation
All six phases implemented in one working tree; measurement in [`measurement.md`](./measurement.md).
| Claim in the plan | Found | Effect |
|---|---|---|
| `pron` → đại từ in the label map | `pron` is *pronunciation* on this wiki (36,533 legacy + 5,501 new headings); pronoun is `pronoun`/`per-pronoun` | map corrected; `pron` is a non-POS section code |
| New dialect headings are `{{ĐM|code}}` | also `{{section|code}}` (3,506) and shorthand codes `n`/`v` | both forms and both shorthands mapped |
| `{{-dfn-}}` is a heading | it is a "definitions" marker placed under a POS heading | transparent to the label |
| Bot's bzip2 may take > 2 min | 21–25 s for the whole build | no fallback |
| Empty-label senses under 10% | 11.9%, all from pages with no POS heading at all; every code seen more than once is mapped | accepted, recorded |
| Fixture bzip2 lives at `testdata/mini-dump.xml.bz2` | placed in the package's own `server/cmd/build-dictionary/testdata/` (Go convention; the repo-root `testdata/` is the Makefile's and e2e's) | path only |
| `expanded: Set<string>` in the store | a `string[]` with set semantics, because Svelte 5's `$state` proxies arrays and not Sets | same rules, same tests |
| Stripper teaches only `label`/`l`/`place` families | also `nhãn`/`context`/`term` (label spellings), `n-g`, and `see-entry`/`like-entry` → `Xem x`, the commonest definition-line templates | phase-1 risk response, recorded |
| First build lost 15 kaikki words | pages with `{{-vie-}}{{-pron-}}…{{-place-}}` on one line | scanner reads several headings per line; second build lost 0 |
Numbers: 36,200 words (34,813 shared with kaikki, 1,387 gained, 0 lost), 40,842 senses on
35,062 words (96.9%), 43,011 pages, no graph metric below the kaikki database's.
Verification: `go vet ./... && go test ./... -race` green; `npm run check && npm test` green
(183); `npx playwright test` 38/39 on each full run, the one failure being the four-player
seating spec ("a fifth is turned away"), which no changed file touches and which fails about
one run in three on this machine with local Chrome **on `HEAD` as well** (stashed tree,
`--repeat-each 4`: 3 passed, 1 failed) — a pre-existing flake, not a regression; both meanings
specs pass every run; Docker image variants
**not built locally** — the daemon was not running — and left to CI. `--min-pages` (default
20,000) was added to the builder so the page floor is a flag like `--min-words`, which the
tests need to lower.
Code review (`plans/reports/code-reviewer-260908-2210-…md`): no critical finding; one high
(a flaky e2e assertion on the opening's label), three medium (nested-list styling leak,
format characters surviving the stripper, opaque refusal of a v4 database) and eight low, all
fixed in the same tree and re-verified; see `measurement.md` "After code review".
<!-- slug: wiktionary-dump-corpus-and-meanings -->