noitu/plans/260908-2056-wiktionary-dump-corpus-and-meanings/phase-02-fetch-the-dump.md

phase, title, status, priority, effort, dependencies

phase	title	status	priority	effort	dependencies
2	Phase 2: Fetch the dump	completed	P1	2h	1

Phase 2: Fetch the dump

Overview

Point the Makefile, the Docker dict stage, the pin test, the CI leak guard and the README's raw commands at Wikimedia's rolling latest dump instead of kaikki's JSONL, keeping the three URL copies in agreement and a failed download unable to pass as a success.

Requirements

• Functional: DICT_URL is https://dumps.wikimedia.org/viwiktionary/latest/viwiktionary-latest-pages-articles.xml.bz2; DICT_SRC is data/viwiktionary-latest-pages-articles.xml.bz2. fetch-dict uses curl -fLR — -R keeps the server's modification time, which is the dump's generation time and becomes source_fetched_at — to a .part name renamed on success, as today.
• Functional: dict runs build-dictionary --dump ../$(DICT_SRC).
• Functional: the Dockerfile dict stage mirrors the URL and the flag; FIXTURE_DICT=1 is untouched. curl gains -R there too.
• Functional: web/tests/dictionary-source.test.js asserts the Makefile and Dockerfile URLs agree, that the URL matches ^https://dumps\.wikimedia\.org/viwiktionary/latest/viwiktionary-latest-pages-articles\.xml\.bz2$, that the builder's dumpSourceURL constant in dump.go is the same string, and that README and data/ATTRIBUTION.md quote it.
• Functional: the CI leak guard rejects any \.bz2$ or \.xml$ inside the image; the comment says the dump, not the export. .gitignore covers data/*.bz2, data/*.bz2.part, data/*.xml (for the phase-1 fallback) and drops the .jsonl lines.
• Functional: README's "Without make" block, Make targets table and Setup section describe a ~61 MB monthly dump; the fetch-dict help line says the same.
• Non-functional: still no resume flag. latest can be repointed between two attempts.

Architecture

DICT_URL := https://dumps.wikimedia.org/viwiktionary/latest/viwiktionary-latest-pages-articles.xml.bz2
DICT_SRC := data/viwiktionary-latest-pages-articles.xml.bz2

fetch-dict:
	@mkdir -p data
	curl -fLR -o $(DICT_SRC).part $(DICT_URL) && mv $(DICT_SRC).part $(DICT_SRC)

dict: $(DICT_SRC)
	cd server && go run ./cmd/build-dictionary --dump ../$(DICT_SRC) --out ../$(DICT_OUT)

Correctness rests on: curl -f plus the .part rename (HTTP errors, interrupted downloads); the bzip2 decoder (a truncated stream errors at EOF); the XML decoder (a cut mid-page); the 20,000-page check and the 30,000-word floor (content).

Related Code Files

• Modify: Makefile — header comment, DICT_URL, DICT_SRC, help, fetch-dict, dict
• Modify: Dockerfile — dict stage comment, ARG DICT_URL, curl -fsSLR, file name, --dump
• Modify: web/tests/dictionary-source.test.js — URL pattern, builder file and constant name
• Modify: .github/workflows/ci.yml — leak guard pattern and comment; the image job's comment about "the real upstream release" still holds
• Modify: .gitignore
• Modify: README.md — Setup, Make targets, Without make, Architecture table's dictionary row wording if it names the export

Implementation Steps

1. Update the Makefile; run make fetch-dict on a clean data/ and check the file's mtime is the dump's, not now.
2. Mirror in the Dockerfile.
3. Rewrite the pin test; run npx vitest run tests/dictionary-source.test.js. Edit one URL copy alone and confirm it fails.
4. Update the CI leak guard, .gitignore, README.
5. make dict; then docker build . and docker build --build-arg FIXTURE_DICT=1 .; run the CI file checks against the real image and confirm no .bz2/.xml inside.
6. Point DICT_URL at a 404 once and confirm fetch-dict fails; feed dict a copy of the dump cut at 40 MB and confirm the build fails naming the cause.

Success Criteria

• make fetch-dict downloads ~61 MB with the server's mtime; make dict builds >30,000 words with meanings.
• web/tests/dictionary-source.test.js passes and fails when any one copy is edited.
• Both image variants build; the real one passes the CI file checks; nothing matching \.bz2$|\.xml$ inside.
• grep -rniI kaikki --exclude-dir=plans . finds nothing outside plans/.
• A 404 URL and a truncated file each fail the build readably.

Risk Assessment

latest/ mid-repoint. Wikimedia updates the latest symlinks file by file as a run completes. Fetching during the run can return the previous month's file, which is correct, or in principle a 404 for a moment. Signal: curl: (22). Response: retry; nothing to fix.

Dump size grows. Irrelevant to correctness; the README's "~61 MB" goes stale slowly. Signal: none. Response: update the number when it is off by a quarter.

The Docker build now decompresses bzip2 in Go inside alpine. Same code, same speed as locally. If phase 1 chose the bzip2 -dc fallback, the stage needs apk add bzip2 and the pipe; the leak guard's .xml clause is why an uncompressed intermediate is also checked.