mirror of
https://github.com/tiennm99/noitu.git
synced 2026-10-11 03:13:45 +00:00
105 lines
4.9 KiB
Markdown
105 lines
4.9 KiB
Markdown
---
|
|
phase: 2
|
|
title: "Phase 2: Fetch the dump"
|
|
status: completed
|
|
priority: P1
|
|
effort: "2h"
|
|
dependencies: [1]
|
|
---
|
|
|
|
# Phase 2: Fetch the dump
|
|
|
|
## Overview
|
|
|
|
Point the Makefile, the Docker `dict` stage, the pin test, the CI leak guard and the README's
|
|
raw commands at Wikimedia's rolling `latest` dump instead of kaikki's JSONL, keeping the
|
|
three URL copies in agreement and a failed download unable to pass as a success.
|
|
|
|
## Requirements
|
|
|
|
- Functional: `DICT_URL` is
|
|
`https://dumps.wikimedia.org/viwiktionary/latest/viwiktionary-latest-pages-articles.xml.bz2`;
|
|
`DICT_SRC` is `data/viwiktionary-latest-pages-articles.xml.bz2`. `fetch-dict` uses
|
|
`curl -fLR` — `-R` keeps the server's modification time, which is the dump's generation
|
|
time and becomes `source_fetched_at` — to a `.part` name renamed on success, as today.
|
|
- Functional: `dict` runs `build-dictionary --dump ../$(DICT_SRC)`.
|
|
- Functional: the Dockerfile `dict` stage mirrors the URL and the flag; `FIXTURE_DICT=1` is
|
|
untouched. `curl` gains `-R` there too.
|
|
- Functional: `web/tests/dictionary-source.test.js` asserts the Makefile and Dockerfile URLs
|
|
agree, that the URL matches
|
|
`^https://dumps\.wikimedia\.org/viwiktionary/latest/viwiktionary-latest-pages-articles\.xml\.bz2$`,
|
|
that the builder's `dumpSourceURL` constant in `dump.go` is the same string, and that README
|
|
and `data/ATTRIBUTION.md` quote it.
|
|
- Functional: the CI leak guard rejects any `\.bz2$` or `\.xml$` inside the image; the
|
|
comment says the dump, not the export. `.gitignore` covers `data/*.bz2`, `data/*.bz2.part`,
|
|
`data/*.xml` (for the phase-1 fallback) and drops the `.jsonl` lines.
|
|
- Functional: README's "Without make" block, Make targets table and Setup section describe
|
|
a ~61 MB monthly dump; the `fetch-dict` help line says the same.
|
|
- Non-functional: still no resume flag. `latest` can be repointed between two attempts.
|
|
|
|
## Architecture
|
|
|
|
```make
|
|
DICT_URL := https://dumps.wikimedia.org/viwiktionary/latest/viwiktionary-latest-pages-articles.xml.bz2
|
|
DICT_SRC := data/viwiktionary-latest-pages-articles.xml.bz2
|
|
|
|
fetch-dict:
|
|
@mkdir -p data
|
|
curl -fLR -o $(DICT_SRC).part $(DICT_URL) && mv $(DICT_SRC).part $(DICT_SRC)
|
|
|
|
dict: $(DICT_SRC)
|
|
cd server && go run ./cmd/build-dictionary --dump ../$(DICT_SRC) --out ../$(DICT_OUT)
|
|
```
|
|
|
|
Correctness rests on: `curl -f` plus the `.part` rename (HTTP errors, interrupted
|
|
downloads); the bzip2 decoder (a truncated stream errors at EOF); the XML decoder (a cut
|
|
mid-page); the 20,000-page check and the 30,000-word floor (content).
|
|
|
|
## Related Code Files
|
|
|
|
- Modify: `Makefile` — header comment, `DICT_URL`, `DICT_SRC`, `help`, `fetch-dict`, `dict`
|
|
- Modify: `Dockerfile` — `dict` stage comment, `ARG DICT_URL`, `curl -fsSLR`, file name,
|
|
`--dump`
|
|
- Modify: `web/tests/dictionary-source.test.js` — URL pattern, builder file and constant name
|
|
- Modify: `.github/workflows/ci.yml` — leak guard pattern and comment; the `image` job's
|
|
comment about "the real upstream release" still holds
|
|
- Modify: `.gitignore`
|
|
- Modify: `README.md` — Setup, Make targets, Without make, Architecture table's dictionary
|
|
row wording if it names the export
|
|
|
|
## Implementation Steps
|
|
|
|
1. Update the Makefile; run `make fetch-dict` on a clean `data/` and check the file's mtime is
|
|
the dump's, not now.
|
|
2. Mirror in the Dockerfile.
|
|
3. Rewrite the pin test; run `npx vitest run tests/dictionary-source.test.js`. Edit one URL
|
|
copy alone and confirm it fails.
|
|
4. Update the CI leak guard, `.gitignore`, README.
|
|
5. `make dict`; then `docker build .` and `docker build --build-arg FIXTURE_DICT=1 .`; run
|
|
the CI file checks against the real image and confirm no `.bz2`/`.xml` inside.
|
|
6. Point `DICT_URL` at a 404 once and confirm `fetch-dict` fails; feed `dict` a copy of the
|
|
dump cut at 40 MB and confirm the build fails naming the cause.
|
|
|
|
## Success Criteria
|
|
|
|
- [x] `make fetch-dict` downloads ~61 MB with the server's mtime; `make dict` builds >30,000
|
|
words with meanings.
|
|
- [x] `web/tests/dictionary-source.test.js` passes and fails when any one copy is edited.
|
|
- [x] Both image variants build; the real one passes the CI file checks; nothing matching
|
|
`\.bz2$|\.xml$` inside.
|
|
- [x] `grep -rniI kaikki --exclude-dir=plans .` finds nothing outside `plans/`.
|
|
- [x] A 404 URL and a truncated file each fail the build readably.
|
|
|
|
## Risk Assessment
|
|
|
|
**`latest/` mid-repoint.** Wikimedia updates the `latest` symlinks file by file as a run
|
|
completes. Fetching during the run can return the previous month's file, which is correct,
|
|
or in principle a 404 for a moment. Signal: `curl: (22)`. Response: retry; nothing to fix.
|
|
|
|
**Dump size grows.** Irrelevant to correctness; the README's "~61 MB" goes stale slowly.
|
|
Signal: none. Response: update the number when it is off by a quarter.
|
|
|
|
**The Docker build now decompresses bzip2 in Go inside alpine.** Same code, same speed as
|
|
locally. If phase 1 chose the `bzip2 -dc` fallback, the stage needs `apk add bzip2` and the
|
|
pipe; the leak guard's `.xml` clause is why an uncompressed intermediate is also checked.
|