ci(refresh-dictionary): commit the refreshed corpus straight to main

This commit is contained in:
tiennm99 committed 2026-10-10 17:29:24 +07:00
1 parent 4d12e283c6
commit 077cc38d5f
3 files changed
+33 -39

No files matched your search

+26 -33
View File
@@ -2,16 +2,17 @@
# #
# Wikimedia regenerates the Wiktionary tiếng Việt dump every month. This # Wikimedia regenerates the Wiktionary tiếng Việt dump every month. This
# downloads the current one, regenerates data/dictionary.txt from it — the # downloads the current one, regenerates data/dictionary.txt from it — the
# committed corpus the image is built from — and opens a pull request against # committed corpus the image is built from — and commits it straight to main,
# main, so a new month's words arrive as a diff someone reads before it ships. # which deploys it. The commit is the month's diff of the word list.
# #
# The dump run that starts on the 1st takes a few days to publish, hence the # The dump run that starts on the 1st takes a few days to publish, hence the
# 10th. The builder fails below its word, page and meaning-coverage floors, so # 10th. The builder fails below its word, page and meaning-coverage floors, so
# a broken month opens no pull request rather than a corpus with holes in it. # a broken month commits nothing rather than a corpus with holes in it. A month
# whose dump matches the committed corpus commits nothing either.
# #
# A pull request opened with GITHUB_TOKEN does not trigger ci.yml. The build # A push made with GITHUB_TOKEN does not trigger ci.yml. The build here
# here already verifies the database it derives; the full suite runs when the # already verifies the database it derives; the full suite runs on the next
# pull request is merged. # push to main.
name: refresh-dictionary name: refresh-dictionary
on: on:
@@ -21,7 +22,6 @@ on:
permissions: permissions:
contents: write contents: write
pull-requests: write
concurrency: concurrency:
group: refresh-dictionary group: refresh-dictionary
@@ -46,37 +46,30 @@ jobs:
- name: Regenerate the corpus - name: Regenerate the corpus
run: make refresh-dict run: make refresh-dict
# What the pull request body reports: the dump's date and hash from the # The commit message reports the dump's date and hash from the corpus
# corpus header, and how many words came and went. # header, and how many words came and went.
- name: Summarize the change - name: Commit and push
id: summary
run: | run: |
set -eu set -eu
if git diff --quiet -- data/dictionary.txt; then
echo "The corpus matches the current dump; nothing to commit."
exit 0
fi
header() { sed -n "s/^#@$1 //p" data/dictionary.txt; } header() { sed -n "s/^#@$1 //p" data/dictionary.txt; }
fetched="$(header source_fetched_at)" fetched="$(header source_fetched_at)"
added="$(git diff -U0 -- data/dictionary.txt | grep -c '^+[^+#]' || true)" added="$(git diff -U0 -- data/dictionary.txt | grep -c '^+[^+#]' || true)"
removed="$(git diff -U0 -- data/dictionary.txt | grep -c '^-[^-#]' || true)" removed="$(git diff -U0 -- data/dictionary.txt | grep -c '^-[^-#]' || true)"
words="$(grep -vc '^#\|^$' data/dictionary.txt)" words="$(grep -vc '^#\|^$' data/dictionary.txt)"
{ git config user.name 'github-actions[bot]'
echo "date=${fetched%%T*}" git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
echo "body<<EOF" git add data/dictionary.txt
echo "Regenerated \`data/dictionary.txt\` from the Wiktionary tiếng Việt dump of ${fetched}." git commit -F - <<MSG
echo chore(data): refresh the dictionary from the ${fetched%%T*} dump
echo "- Dump SHA-256: \`$(header source_sha256)\`"
echo "- Words in the corpus: ${words}"
echo "- Lines added: ${added}, lines removed: ${removed} (a word whose meanings changed counts as both)"
echo
echo "Opened by the monthly refresh-dictionary workflow. CI does not run on this pull request; the builder verified the database before exporting the corpus."
echo "EOF"
} >> "$GITHUB_OUTPUT"
- name: Open the pull request Regenerated data/dictionary.txt from the Wiktionary tiếng Việt dump of ${fetched}.
uses: peter-evans/create-pull-request@v8
with: - Dump SHA-256: $(header source_sha256)
base: main - Words in the corpus: ${words}
branch: dictionary-refresh - Lines added: ${added}, lines removed: ${removed} (a word whose meanings changed counts as both)
delete-branch: true MSG
add-paths: data/dictionary.txt git push origin HEAD:main
commit-message: "chore(data): refresh the dictionary from the ${{ steps.summary.outputs.date }} dump"
title: "chore(data): refresh the dictionary from the ${{ steps.summary.outputs.date }} dump"
body: ${{ steps.summary.outputs.body }}
+1 -1
View File
@@ -185,7 +185,7 @@ The dictionary is committed as text: [`data/dictionary.txt`](./data/dictionary.t
every word the builder accepted from the Wiktionary tiếng Việt dump, one per line with its every word the builder accepted from the Wiktionary tiếng Việt dump, one per line with its
meanings, and a header naming the dump and its SHA-256. Wikimedia regenerates the dump meanings, and a header naming the dump and its SHA-256. Wikimedia regenerates the dump
monthly, and the [`refresh-dictionary`](./.github/workflows/refresh-dictionary.yml) workflow monthly, and the [`refresh-dictionary`](./.github/workflows/refresh-dictionary.yml) workflow
opens a pull request against `main` with the new month's corpus on the 10th, so every change commits the new month's corpus to `main` on the 10th, so every change
to the word list is a reviewable diff. To refresh by hand, `make fetch-dict` (downloads the to the word list is a reviewable diff. To refresh by hand, `make fetch-dict` (downloads the
~61 MB dump) then `make refresh-dict`. Neither the dump nor `data/noitu.db` is committed. See ~61 MB dump) then `make refresh-dict`. Neither the dump nor `data/noitu.db` is committed. See
[`data/ATTRIBUTION.md`](./data/ATTRIBUTION.md). [`data/ATTRIBUTION.md`](./data/ATTRIBUTION.md).
+6 -5
View File
@@ -391,12 +391,13 @@ the dump it came from.
[`refresh-dictionary.yml`](../.github/workflows/refresh-dictionary.yml) [`refresh-dictionary.yml`](../.github/workflows/refresh-dictionary.yml)
downloads the current dump on the 10th of every month, or on demand from the downloads the current dump on the 10th of every month, or on demand from the
Actions tab, regenerates the corpus and opens a pull request against `main`. Actions tab, regenerates the corpus and commits it straight to `main`, which
Merging it deploys the new words like any other change. The pull request does not trigger CI, because GitHub does not run deploys the new words like any other change. A month whose dump matches the
workflows on a pull request opened with `GITHUB_TOKEN`; the builder verifies committed corpus commits nothing. The push does not trigger CI, because GitHub
the database before exporting the corpus, and CI runs again on the merge. does not run workflows on a push made with `GITHUB_TOKEN`; the builder verifies
the database before exporting the corpus, and CI runs again on the next push.
GitHub disables scheduled workflows after 60 days without a commit to the GitHub disables scheduled workflows after 60 days without a commit to the
repository, so re-enable it from the Actions tab if the pull requests stop. repository, so re-enable it from the Actions tab if the monthly commits stop.
To refresh by hand, run `make fetch-dict` and `make refresh-dict`, then commit To refresh by hand, run `make fetch-dict` and `make refresh-dict`, then commit
`data/dictionary.txt`. `data/dictionary.txt`.