mirror of
https://github.com/tiennm99/noitu.git
synced 2026-10-11 03:13:45 +00:00
ci(refresh-dictionary): commit the refreshed corpus straight to main
This commit is contained in:
1 parent
4d12e283c6
commit
077cc38d5f
3 files changed
+33
-39
No files matched your search
@@ -2,16 +2,17 @@
|
|||||||
#
|
#
|
||||||
# Wikimedia regenerates the Wiktionary tiếng Việt dump every month. This
|
# Wikimedia regenerates the Wiktionary tiếng Việt dump every month. This
|
||||||
# downloads the current one, regenerates data/dictionary.txt from it — the
|
# downloads the current one, regenerates data/dictionary.txt from it — the
|
||||||
# committed corpus the image is built from — and opens a pull request against
|
# committed corpus the image is built from — and commits it straight to main,
|
||||||
# main, so a new month's words arrive as a diff someone reads before it ships.
|
# which deploys it. The commit is the month's diff of the word list.
|
||||||
#
|
#
|
||||||
# The dump run that starts on the 1st takes a few days to publish, hence the
|
# The dump run that starts on the 1st takes a few days to publish, hence the
|
||||||
# 10th. The builder fails below its word, page and meaning-coverage floors, so
|
# 10th. The builder fails below its word, page and meaning-coverage floors, so
|
||||||
# a broken month opens no pull request rather than a corpus with holes in it.
|
# a broken month commits nothing rather than a corpus with holes in it. A month
|
||||||
|
# whose dump matches the committed corpus commits nothing either.
|
||||||
#
|
#
|
||||||
# A pull request opened with GITHUB_TOKEN does not trigger ci.yml. The build
|
# A push made with GITHUB_TOKEN does not trigger ci.yml. The build here
|
||||||
# here already verifies the database it derives; the full suite runs when the
|
# already verifies the database it derives; the full suite runs on the next
|
||||||
# pull request is merged.
|
# push to main.
|
||||||
name: refresh-dictionary
|
name: refresh-dictionary
|
||||||
|
|
||||||
on:
|
on:
|
||||||
@@ -21,7 +22,6 @@ on:
|
|||||||
|
|
||||||
permissions:
|
permissions:
|
||||||
contents: write
|
contents: write
|
||||||
pull-requests: write
|
|
||||||
|
|
||||||
concurrency:
|
concurrency:
|
||||||
group: refresh-dictionary
|
group: refresh-dictionary
|
||||||
@@ -46,37 +46,30 @@ jobs:
|
|||||||
- name: Regenerate the corpus
|
- name: Regenerate the corpus
|
||||||
run: make refresh-dict
|
run: make refresh-dict
|
||||||
|
|
||||||
# What the pull request body reports: the dump's date and hash from the
|
# The commit message reports the dump's date and hash from the corpus
|
||||||
# corpus header, and how many words came and went.
|
# header, and how many words came and went.
|
||||||
- name: Summarize the change
|
- name: Commit and push
|
||||||
id: summary
|
|
||||||
run: |
|
run: |
|
||||||
set -eu
|
set -eu
|
||||||
|
if git diff --quiet -- data/dictionary.txt; then
|
||||||
|
echo "The corpus matches the current dump; nothing to commit."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
header() { sed -n "s/^#@$1 //p" data/dictionary.txt; }
|
header() { sed -n "s/^#@$1 //p" data/dictionary.txt; }
|
||||||
fetched="$(header source_fetched_at)"
|
fetched="$(header source_fetched_at)"
|
||||||
added="$(git diff -U0 -- data/dictionary.txt | grep -c '^+[^+#]' || true)"
|
added="$(git diff -U0 -- data/dictionary.txt | grep -c '^+[^+#]' || true)"
|
||||||
removed="$(git diff -U0 -- data/dictionary.txt | grep -c '^-[^-#]' || true)"
|
removed="$(git diff -U0 -- data/dictionary.txt | grep -c '^-[^-#]' || true)"
|
||||||
words="$(grep -vc '^#\|^$' data/dictionary.txt)"
|
words="$(grep -vc '^#\|^$' data/dictionary.txt)"
|
||||||
{
|
git config user.name 'github-actions[bot]'
|
||||||
echo "date=${fetched%%T*}"
|
git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
|
||||||
echo "body<<EOF"
|
git add data/dictionary.txt
|
||||||
echo "Regenerated \`data/dictionary.txt\` from the Wiktionary tiếng Việt dump of ${fetched}."
|
git commit -F - <<MSG
|
||||||
echo
|
chore(data): refresh the dictionary from the ${fetched%%T*} dump
|
||||||
echo "- Dump SHA-256: \`$(header source_sha256)\`"
|
|
||||||
echo "- Words in the corpus: ${words}"
|
|
||||||
echo "- Lines added: ${added}, lines removed: ${removed} (a word whose meanings changed counts as both)"
|
|
||||||
echo
|
|
||||||
echo "Opened by the monthly refresh-dictionary workflow. CI does not run on this pull request; the builder verified the database before exporting the corpus."
|
|
||||||
echo "EOF"
|
|
||||||
} >> "$GITHUB_OUTPUT"
|
|
||||||
|
|
||||||
- name: Open the pull request
|
Regenerated data/dictionary.txt from the Wiktionary tiếng Việt dump of ${fetched}.
|
||||||
uses: peter-evans/create-pull-request@v8
|
|
||||||
with:
|
- Dump SHA-256: $(header source_sha256)
|
||||||
base: main
|
- Words in the corpus: ${words}
|
||||||
branch: dictionary-refresh
|
- Lines added: ${added}, lines removed: ${removed} (a word whose meanings changed counts as both)
|
||||||
delete-branch: true
|
MSG
|
||||||
add-paths: data/dictionary.txt
|
git push origin HEAD:main
|
||||||
commit-message: "chore(data): refresh the dictionary from the ${{ steps.summary.outputs.date }} dump"
|
|
||||||
title: "chore(data): refresh the dictionary from the ${{ steps.summary.outputs.date }} dump"
|
|
||||||
body: ${{ steps.summary.outputs.body }}
|
|
||||||
@@ -185,7 +185,7 @@ The dictionary is committed as text: [`data/dictionary.txt`](./data/dictionary.t
|
|||||||
every word the builder accepted from the Wiktionary tiếng Việt dump, one per line with its
|
every word the builder accepted from the Wiktionary tiếng Việt dump, one per line with its
|
||||||
meanings, and a header naming the dump and its SHA-256. Wikimedia regenerates the dump
|
meanings, and a header naming the dump and its SHA-256. Wikimedia regenerates the dump
|
||||||
monthly, and the [`refresh-dictionary`](./.github/workflows/refresh-dictionary.yml) workflow
|
monthly, and the [`refresh-dictionary`](./.github/workflows/refresh-dictionary.yml) workflow
|
||||||
opens a pull request against `main` with the new month's corpus on the 10th, so every change
|
commits the new month's corpus to `main` on the 10th, so every change
|
||||||
to the word list is a reviewable diff. To refresh by hand, `make fetch-dict` (downloads the
|
to the word list is a reviewable diff. To refresh by hand, `make fetch-dict` (downloads the
|
||||||
~61 MB dump) then `make refresh-dict`. Neither the dump nor `data/noitu.db` is committed. See
|
~61 MB dump) then `make refresh-dict`. Neither the dump nor `data/noitu.db` is committed. See
|
||||||
[`data/ATTRIBUTION.md`](./data/ATTRIBUTION.md).
|
[`data/ATTRIBUTION.md`](./data/ATTRIBUTION.md).
|
||||||
|
|||||||
+6
-5
@@ -391,12 +391,13 @@ the dump it came from.
|
|||||||
|
|
||||||
[`refresh-dictionary.yml`](../.github/workflows/refresh-dictionary.yml)
|
[`refresh-dictionary.yml`](../.github/workflows/refresh-dictionary.yml)
|
||||||
downloads the current dump on the 10th of every month, or on demand from the
|
downloads the current dump on the 10th of every month, or on demand from the
|
||||||
Actions tab, regenerates the corpus and opens a pull request against `main`.
|
Actions tab, regenerates the corpus and commits it straight to `main`, which
|
||||||
Merging it deploys the new words like any other change. The pull request does not trigger CI, because GitHub does not run
|
deploys the new words like any other change. A month whose dump matches the
|
||||||
workflows on a pull request opened with `GITHUB_TOKEN`; the builder verifies
|
committed corpus commits nothing. The push does not trigger CI, because GitHub
|
||||||
the database before exporting the corpus, and CI runs again on the merge.
|
does not run workflows on a push made with `GITHUB_TOKEN`; the builder verifies
|
||||||
|
the database before exporting the corpus, and CI runs again on the next push.
|
||||||
GitHub disables scheduled workflows after 60 days without a commit to the
|
GitHub disables scheduled workflows after 60 days without a commit to the
|
||||||
repository, so re-enable it from the Actions tab if the pull requests stop.
|
repository, so re-enable it from the Actions tab if the monthly commits stop.
|
||||||
|
|
||||||
To refresh by hand, run `make fetch-dict` and `make refresh-dict`, then commit
|
To refresh by hand, run `make fetch-dict` and `make refresh-dict`, then commit
|
||||||
`data/dictionary.txt`.
|
`data/dictionary.txt`.
|
||||||
|
|||||||
Reference in new issue
Block a user