mirror of
https://github.com/tiennm99/noitu.git
synced 2026-10-11 03:13:45 +00:00
ci(refresh-dictionary): commit the refreshed corpus straight to main
This commit is contained in:
tiennm99 committed
1 parent
4d12e283c6
commit
077cc38d5f
3 files changed
+33
-39
No files matched your search
@@ -2,16 +2,17 @@
|
||||
#
|
||||
# Wikimedia regenerates the Wiktionary tiếng Việt dump every month. This
|
||||
# downloads the current one, regenerates data/dictionary.txt from it — the
|
||||
# committed corpus the image is built from — and opens a pull request against
|
||||
# main, so a new month's words arrive as a diff someone reads before it ships.
|
||||
# committed corpus the image is built from — and commits it straight to main,
|
||||
# which deploys it. The commit is the month's diff of the word list.
|
||||
#
|
||||
# The dump run that starts on the 1st takes a few days to publish, hence the
|
||||
# 10th. The builder fails below its word, page and meaning-coverage floors, so
|
||||
# a broken month opens no pull request rather than a corpus with holes in it.
|
||||
# a broken month commits nothing rather than a corpus with holes in it. A month
|
||||
# whose dump matches the committed corpus commits nothing either.
|
||||
#
|
||||
# A pull request opened with GITHUB_TOKEN does not trigger ci.yml. The build
|
||||
# here already verifies the database it derives; the full suite runs when the
|
||||
# pull request is merged.
|
||||
# A push made with GITHUB_TOKEN does not trigger ci.yml. The build here
|
||||
# already verifies the database it derives; the full suite runs on the next
|
||||
# push to main.
|
||||
name: refresh-dictionary
|
||||
|
||||
on:
|
||||
@@ -21,7 +22,6 @@ on:
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
|
||||
concurrency:
|
||||
group: refresh-dictionary
|
||||
@@ -46,37 +46,30 @@ jobs:
|
||||
- name: Regenerate the corpus
|
||||
run: make refresh-dict
|
||||
|
||||
# What the pull request body reports: the dump's date and hash from the
|
||||
# corpus header, and how many words came and went.
|
||||
- name: Summarize the change
|
||||
id: summary
|
||||
# The commit message reports the dump's date and hash from the corpus
|
||||
# header, and how many words came and went.
|
||||
- name: Commit and push
|
||||
run: |
|
||||
set -eu
|
||||
if git diff --quiet -- data/dictionary.txt; then
|
||||
echo "The corpus matches the current dump; nothing to commit."
|
||||
exit 0
|
||||
fi
|
||||
header() { sed -n "s/^#@$1 //p" data/dictionary.txt; }
|
||||
fetched="$(header source_fetched_at)"
|
||||
added="$(git diff -U0 -- data/dictionary.txt | grep -c '^+[^+#]' || true)"
|
||||
removed="$(git diff -U0 -- data/dictionary.txt | grep -c '^-[^-#]' || true)"
|
||||
words="$(grep -vc '^#\|^$' data/dictionary.txt)"
|
||||
{
|
||||
echo "date=${fetched%%T*}"
|
||||
echo "body<<EOF"
|
||||
echo "Regenerated \`data/dictionary.txt\` from the Wiktionary tiếng Việt dump of ${fetched}."
|
||||
echo
|
||||
echo "- Dump SHA-256: \`$(header source_sha256)\`"
|
||||
echo "- Words in the corpus: ${words}"
|
||||
echo "- Lines added: ${added}, lines removed: ${removed} (a word whose meanings changed counts as both)"
|
||||
echo
|
||||
echo "Opened by the monthly refresh-dictionary workflow. CI does not run on this pull request; the builder verified the database before exporting the corpus."
|
||||
echo "EOF"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
git config user.name 'github-actions[bot]'
|
||||
git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
|
||||
git add data/dictionary.txt
|
||||
git commit -F - <<MSG
|
||||
chore(data): refresh the dictionary from the ${fetched%%T*} dump
|
||||
|
||||
- name: Open the pull request
|
||||
uses: peter-evans/create-pull-request@v8
|
||||
with:
|
||||
base: main
|
||||
branch: dictionary-refresh
|
||||
delete-branch: true
|
||||
add-paths: data/dictionary.txt
|
||||
commit-message: "chore(data): refresh the dictionary from the ${{ steps.summary.outputs.date }} dump"
|
||||
title: "chore(data): refresh the dictionary from the ${{ steps.summary.outputs.date }} dump"
|
||||
body: ${{ steps.summary.outputs.body }}
|
||||
Regenerated data/dictionary.txt from the Wiktionary tiếng Việt dump of ${fetched}.
|
||||
|
||||
- Dump SHA-256: $(header source_sha256)
|
||||
- Words in the corpus: ${words}
|
||||
- Lines added: ${added}, lines removed: ${removed} (a word whose meanings changed counts as both)
|
||||
MSG
|
||||
git push origin HEAD:main
|
||||
@@ -185,7 +185,7 @@ The dictionary is committed as text: [`data/dictionary.txt`](./data/dictionary.t
|
||||
every word the builder accepted from the Wiktionary tiếng Việt dump, one per line with its
|
||||
meanings, and a header naming the dump and its SHA-256. Wikimedia regenerates the dump
|
||||
monthly, and the [`refresh-dictionary`](./.github/workflows/refresh-dictionary.yml) workflow
|
||||
opens a pull request against `main` with the new month's corpus on the 10th, so every change
|
||||
commits the new month's corpus to `main` on the 10th, so every change
|
||||
to the word list is a reviewable diff. To refresh by hand, `make fetch-dict` (downloads the
|
||||
~61 MB dump) then `make refresh-dict`. Neither the dump nor `data/noitu.db` is committed. See
|
||||
[`data/ATTRIBUTION.md`](./data/ATTRIBUTION.md).
|
||||
|
||||
+6
-5
@@ -391,12 +391,13 @@ the dump it came from.
|
||||
|
||||
[`refresh-dictionary.yml`](../.github/workflows/refresh-dictionary.yml)
|
||||
downloads the current dump on the 10th of every month, or on demand from the
|
||||
Actions tab, regenerates the corpus and opens a pull request against `main`.
|
||||
Merging it deploys the new words like any other change. The pull request does not trigger CI, because GitHub does not run
|
||||
workflows on a pull request opened with `GITHUB_TOKEN`; the builder verifies
|
||||
the database before exporting the corpus, and CI runs again on the merge.
|
||||
Actions tab, regenerates the corpus and commits it straight to `main`, which
|
||||
deploys the new words like any other change. A month whose dump matches the
|
||||
committed corpus commits nothing. The push does not trigger CI, because GitHub
|
||||
does not run workflows on a push made with `GITHUB_TOKEN`; the builder verifies
|
||||
the database before exporting the corpus, and CI runs again on the next push.
|
||||
GitHub disables scheduled workflows after 60 days without a commit to the
|
||||
repository, so re-enable it from the Actions tab if the pull requests stop.
|
||||
repository, so re-enable it from the Actions tab if the monthly commits stop.
|
||||
|
||||
To refresh by hand, run `make fetch-dict` and `make refresh-dict`, then commit
|
||||
`data/dictionary.txt`.
|
||||
|
||||
Reference in new issue
Block a user