Phase 1: Read the dump

Overview

Replace the kaikki JSONL reader in build-dictionary with a streaming reader for the Wikimedia pages-articles.xml.bz2 dump that yields, per Vietnamese-section page, the title for accept() and the stripped definition lines for the meanings table.

Requirements

Functional: --dump <file> reads a bzip2-compressed MediaWiki XML export. Pages with ns != 0 or a <redirect> element are skipped and counted. Every other page's <title> and latest <revision><text> are handed to the section scanner.
Functional: the scanner finds the Vietnamese section in either dialect — {{-vie-}} or a level-2 heading whose text is {{langname|vi}} — and stops at the section's end: for legacy pages, the next {{-xxx-}} template whose code is not in the section-code set; for new pages, the next level-2 heading. Pages with no Vietnamese section are counted, not read. Pages with a section in both dialects are read once, first dialect wins, counted.
Functional: the title goes to accept() unchanged from today. Part of speech labels seen in the section ({{-noun-}}, {{ĐM|noun}}, {{vi-noun}} and kin, pr-noun/place included) are tallied for the log and never filter.
Functional: within the section, every line starting with # (exactly one #, then a space — not #:, #*, ##) is a definition. It is stripped (see Architecture), capped at 200 runes with an ellipsis, and appended to the page's senses up to 5. Empty results are dropped. Senses are kept in page order.
Functional: each sense carries the part of speech of the heading it sits under, as a Vietnamese label from a fixed map keyed by the heading code — the same codes in both dialects ({{-noun-}}, {{ĐM|noun}}, {{vi-noun}} → danh từ). Starting map: noun danh từ, verb động từ, adj tính từ, adv phó từ, pr-noun danh từ riêng, place địa danh, pron đại từ, num số từ, conj liên từ, prep giới từ, intj thán từ, part trợ từ, phrase cụm từ, idiom thành ngữ, prov tục ngữ, abbr viết tắt, prefix tiền tố, suffix hậu tố, char chữ. A definition under a heading outside the map, or before any heading, has an empty label and is kept; unmapped codes are counted for the log. Step 1's frequency list decides what else the map needs. (Validation session 1.)
Functional: two pages whose titles normalize to the same word (Việt Nam and việt nam both being real pages) merge: the word is one entry, senses concatenate in page order and the cap applies to the union.
Functional: provenance is measured during the one streaming pass: SHA-256 of the compressed bytes as read, source_pages (pages with a Vietnamese section, redirects excluded), source_fetched_at from the file's modification time. source_rows is gone.
Functional: the build log reports pages seen, ns0 pages, redirects skipped, pages with a Vietnamese section per dialect, POS tally, definitions kept, definitions dropped as empty after stripping, and the ten commonest template names dropped whole.
Functional: --kaikki and kaikki_list.go and its tests are removed. Exactly one of --dump / --words must be given, same rule as today.
Non-functional: a stream that is not bzip2, XML that ends mid-page, or a <text> element the decoder cannot read is an error naming the page title or byte offset. A count of Vietnamese-section pages under 20,000 is an error naming the count, distinct from the word floor, so a parser that silently misses a dialect is caught before --min-words is.
Non-functional: memory is one page at a time. The XML decoder is used token by token; only title, ns, redirect and text of the current page are held.
Non-functional: wall time under two minutes on the real dump on a laptop. Measure and record; see Risk.

Architecture

--dump <file.xml.bz2>
  │ os.Open → io.TeeReader(f, sha256) → bufio.Reader → compress/bzip2 → xml.NewDecoder
  ▼
for each <page>:
  ns==0, no <redirect>  ─no─► count, skip
  section := vietnameseSection(text)      // dialect-aware slice of the wikitext
  section == ""          ─yes─► count "no Vietnamese section", skip
  word, syllables, reason, ok := accept(title)
  senses := definitions(section)          // []sense{pos, gloss}: stripped, ≤5, ≤200 runes each
  pos tally from section headings
  if ok: words[word] = entry{…}; meanings[word] = merge(meanings[word], senses)

type sense struct { pos, gloss string } lives in wikitext.go; pos is the mapped Vietnamese label or empty.

Files:

server/cmd/build-dictionary/
  dump.go            readDump: bz2+xml streaming, page loop, provenance, dumpSourceURL const
  wikitext.go        vietnameseSection, definitions, stripWikitext, posLabels, posLabelMap
  dump_test.go       byte-exact fixtures: one legacy page, one new-dialect page, a redirect,
                     a foreign-only page, a truncated stream, a non-bzip2 file
  wikitext_test.go   table tests for the stripper and the section boundaries

The section-code set for legacy pages is the list of {{-xxx-}} codes that are headings rather than language switches: etym, pron, noun, verb, adj, adv, pr-noun, place, phrase, idiom, prov, syn, synonym, ant, trans, ref, reference, see, der, info, num, pron, conj, prep, intj, part, abbr, char, hanzi, nom, …. Build it from the real dump in this phase: extract every {{-xxx-}} code, list them by frequency, and classify by hand; the language codes are 2–3 letters and the section codes are English abbreviations, so the split is readable. Record the set in wikitext.go with the frequency it was seen at.

The stripper, in order:

Drop , <ref …>…</ref>, <ref … />, any other tag pair or lone tag.
Templates by a depth counter over {{/}}. For each outermost template, split on | outside nested braces and brackets, take the name:
- label, lb, gloss, qualifier, q → ( join of positional params after a leading vi );
- l, vi-l, w, m, link → the last positional param (display text if any);
- place → positional params after vi, each with a leading [a-z]/ removed, joined by , ;
- anything else → dropped, name counted.
Links: [[Thể loại:…]]/[[Category:…]] dropped; [[a|b]] → b; [[a]] → a; external [http… label] → label.
''', '' removed.   and the common entities decoded.
Control characters (unicode.IsControl) removed; whitespace collapsed; trimmed. A result that is only punctuation is empty.
Longer than 200 runes: cut at the last space before 200 and append ….

Create: server/cmd/build-dictionary/dump.go, wikitext.go, dump_test.go, wikitext_test.go
Delete: server/cmd/build-dictionary/kaikki_list.go, kaikki_list_test.go
Modify: server/cmd/build-dictionary/main.go — package doc, config.kaikki → dump, flag text, run() switch, runFromKaikkiList → runFromDump, entry gains nothing (the meanings map travels beside words into finish() — phase 3 persists it), builderVer = "5"
Modify: server/cmd/build-dictionary/main_test.go — fixtureSource writes a small bzip2 XML dump instead of JSONL (Go has no bzip2 writer: commit a tiny testdata/mini-dump.xml.bz2 built once with bzip2, plus its uncompressed source beside it for review)
Modify: server/cmd/build-dictionary/filter.go — rejectNotVietnamese comment now refers to a page with no Vietnamese section; the reason string can stay

Implementation Steps

Download the current dump once locally (phase 2's URL, curl -fLR). Write a throwaway go run that streams it and prints every {{-xxx-}} code with counts and every level-2 heading text with counts. Build the section-code set and the dialect markers from that output; keep the numbers in comments.
Write wikitext.go: vietnameseSection, posLabels, definitions, stripWikitext. Table-test the stripper on the three samples in plan.md plus: nested templates, a <ref> mid-sentence, a link with a category, a definition that is only {{rfdef|vi}} (must be empty), a 300-rune definition (must end in … at a word boundary), a legacy page whose senses sit under {{-noun-}} then {{-verb-}} (labels danh từ, động từ in order), a new-dialect page with === {{ĐM|pr-noun}} === (label danh từ riêng), a definition under an unmapped heading (empty label, sense kept).
Write dump.go: the streaming loop, provenance, counters, the error shapes listed under Non-functional. Test against testdata/mini-dump.xml.bz2 (six pages, both dialects, a redirect, an English-only page) and byte-truncated copies of it.
Rewire main.go; delete the kaikki files; make go vet ./... && go test ./cmd/... green.
Run against the real dump with --out /tmp/dump.db. Read the log: pages per dialect should be within a few percent of 35,885 / 7,128 for the 2026-09-01 dump; accepted words near 36,200. Record wall time. Keep this database for phase 6.
Sample 50 senses at random from the log (or from the phase-3 table) and read them. Fix any stripper rule that is clearly wrong across many entries; note the rest for phase 6.

Success Criteria

go run ./cmd/build-dictionary --dump ../data/viwiktionary-latest-pages-articles.xml.bz2 --out /tmp/dump.db completes; the log shows ~43,000 Vietnamese-section pages, both dialects non-zero, ~36,200 accepted words, and a definitions-kept count above 30,000.
The stripper table tests pass on all listed shapes; # {{rfdef|vi}} yields no sense; labels come out in Vietnamese for both dialects and empty for an unmapped heading.
On the real dump, senses with an empty label are under 10% of all senses, or the unmapped-code list has been worked through.
A truncated dump, a non-bzip2 file and a mid-page cut each fail with a message naming the cause; a fixture with 3 Vietnamese pages fails the 20,000-page check by name.
grep -rn kaikki server/ finds nothing.
Wall time on the real dump recorded in the phase-6 measurement file.

Risk Assessment

Go's bzip2 is slow. Signal: step 5 takes longer than two minutes. Response: keep the reader on an io.Reader, add --dump acceptance of a plain .xml (sniff the two magic bytes BZ), and have the Makefile pipe bzip2 -dc in; the Docker dict stage on alpine has bzip2. Decide in this phase, not later.

The section-code set is incomplete. A legacy heading code missing from the set is read as a language switch and truncates the Vietnamese section early: definitions after it are lost, the word is not. Signal: the definitions-kept count is well below the page count, or the sample in step 6 shows senses cut off. Response: the step-1 frequency list is the source of truth; every code seen more than ~20 times must be classified.

Definitions in a template the stripper does not know. The whole sense is dropped. Signal: "definitions dropped as empty" is large, or a common template name tops the dropped list. Response: teach the stripper that template if it is a definition-shaped one (place, label are the known cases); otherwise accept the loss and record it in phase 6.

Both dialects on one page. Rare; first dialect wins and the page is counted. Signal: the counter is not small. Response: read both sections and merge senses — a small change to vietnameseSection returning a slice.

12 KiB Raw Blame History Unescape Escape