tiennm99
39ef45730f
Replace the 179 MB minhqnd SQLite aggregate with the 4.8 MB undertheseanlp/dictionary JSONL, pinned by commit and SHA-256, reading only rows tagged "wiktionary". The two other wordlists in that file are never read: hongocduc is GPL and would force a relicense, tudientv is an unlicensed derivative of a commercial dictionary. build-dictionary gains --merged and --sources (names validated, default wiktionary), a shared finish() tail, and meta rows for the source commit and the sources kept and excluded. The SQLite --in path, its schema auto-detection and their tests are removed. Fixture builds now record that they carry no upstream data instead of inheriting a licence string. The --min-words floor moves from 40,000 to 20,000; the corpus is 26,845 words, down from 48,216, all of the loss being words absent from the 2018 Wiktionary scrape. Capitalization is not a filter. The data licence follows the source text: CC BY-SA 3.0 Unported, which is what vi.wiktionary.org carried in 2018. LICENSE, NOTICE, ATTRIBUTION, the README, the image docs, the builder's meta string and the in-game footer all name Wiktionary tiếng Việt's contributors as the authors and undertheseanlp as the intermediary. The Makefile/Dockerfile pin test now also checks the commit the builder stamps into the database.
5.5 KiB
Dictionary Data Attribution
The Vietnamese dictionary data used by this game is not original work of this project. It is derived from Wiktionary text licensed under CC BY-SA 3.0 Unported, and this file records the attribution and the modifications required by that license.
Source
| Field | Value |
|---|---|
| Original work | Entries of Wiktionary tiếng Việt, written by its contributors |
| Original license | CC BY-SA 3.0 Unported at the time of the 2018 scrape (Wikimedia projects moved to 4.0 on 2023-06-29); Wiktionary text is dual-licensed CC BY-SA / GFDL |
| This database's license | CC BY-SA 3.0 Unported, the same licence the source text carried — full text in LICENSE |
| Redistributed by | undertheseanlp/dictionary, a 2018-12-10 scrape of vi.wiktionary.org merged with two other wordlists |
| Asset | dictionary/words.txt at commit 2c078cf — 4,813,111 bytes, 79,226 JSONL rows |
| SHA-256 | 4c3e0e6117e4bdfa97731e135c3d4a05881889909267394a8de8d88ef79f13f0 |
| Rows used | the 32,484 tagged wiktionary |
| Repository code license (not used here) | GPL-3.0 |
The asset is pinned by commit, so the URL is immutable and the checksum above always describes the same bytes. The attribution chain has two links — Wiktionary's contributors, then undertheseanlp as the intermediary that scraped and redistributed — and both are named here because CC BY-SA attribution belongs to the authors, not only to the last host.
Branches deliberately not used
The merged file tags every word with which of three wordlists contain it. Only
wiktionary rows are read; the other two are never consulted — not for words, not for
anything.
hongocduc— Hồ Ngọc Đức's Free Vietnamese Dictionary Project wordlist, distributed under the GNU GPL according to the branch README (a mirror; the canonical site no longer resolves). Using it would make this database GPL-derived and force a relicense to GPLv3. Excluded so the data stays CC BY-SA.tudientv— its branch README declares copyright "Chưa rõ" (unknown) and names the Soha/Vietlex dictionary (Hoàng Phê, editor) as its primary source: an unlicensed derivative of a commercial work. Excluded, and not to be reopened.
Modifications made by this project
server/cmd/build-dictionary transforms the upstream wordlist into data/noitu.db. The
derived database is a modified version of the source data. Changes:
- Source selection — kept only rows whose source list contains
wiktionary; rows from the two other wordlists were not read. - Length filter — kept only words of 2 or more space-separated syllables, as required by the nối từ game rules. Single-syllable entries were dropped.
- Content filter — dropped entries containing digits or punctuation, entries using letters Vietnamese does not have (f, j, w, z), and entries whose syllables do not fit Vietnamese phonotactics (a closed inventory of onsets, nuclei and codas). Diacritic-free Vietnamese words ("con cua") are kept.
- Normalization — all words Unicode NFC-normalized, lowercased, and
whitespace-collapsed. Capitalized headwords (
Hà Nội) become lowercase entries; nothing is removed on the basis of capitalization. - Spelling aliases — added an
aliasestable mapping alternative Vietnamese spellings to canonical entries. Two kinds: competing tone placement in open oa/oe/uy syllables (hoà→hòa,thuý→thúy), and i/y alternation in Sino-Vietnamese syllables (quí→quý,lí→lý). The majority are the i/y kind. These aliases are generated by this project and are not present upstream. - Added columns and tables —
firstandlastsyllable columns, asyllablescount, an index onfirst, asyllablesout-degree table, and ametatable recording provenance (source URL, commit, sources kept and excluded). All added for game lookups. - Deduplication — entries were deduplicated by normalized form, so a word listed both capitalized and lowercase upstream appears once. A generated spelling variant that is itself a real word, or that more than one word would claim, is discarded rather than recorded as an alias.
- Dropped fields — the per-word source tags were discarded after selection. The derived database contains only word forms, not meanings; the upstream wordlist carries none either.
Share-alike obligation
CC BY-SA 3.0 is a share-alike license. The derived database data/noitu.db, and any
distribution of it, remains licensed under CC BY-SA 3.0 — including when it is shipped
inside a container image or any other packaged build of this project.
This obligation applies to the data only. The source code of this project is licensed
separately under Apache-2.0 (see the repository root LICENSE and NOTICE). The derived
database is loaded at runtime from a file and is never compiled or linked into the binary,
keeping the two licensing regimes on separate artifacts.
How to reproduce the derived data
make fetch-dict # downloads the ~4.8 MB upstream wordlist into data/ and verifies its checksum
make dict # derives data/noitu.db from it
Neither file is committed to version control; both are build artifacts.