mirror of
https://github.com/tiennm99/thptqg.git
synced 2026-10-14 08:18:39 +00:00
feat: keep the downloaded database, and stop rebuilding it every deploy
Two caches, one on each side. In the browser, the response is stored in Cache Storage keyed by the server's ETag, so the transfer is paid once per device rather than once per visit. A stored copy opens without the gate: consent was given the first time and reuse costs no network. A redeploy changes the ETag, so the new version replaces the old instead of answering with last week's data, and every older version of that database is dropped so one dataset never occupies the disk twice. When the server cannot be reached at all, any stored version is used, which incidentally makes the site work offline. Storing is best effort — a full disk means downloading again next time, which is no reason to fail the page. The response is cached from a clone while the original is read for the progress bar, rather than buffering the file a second time in the one place where memory is already the binding constraint. In CI, the built databases are cached on the inputs that determine them: data/**, parser/** and datasets.json. Parsing 348 MB of spreadsheets is the slow part of the job, and a web or docs change cannot alter a database, so those pushes restore instead of rebuilding. Exact matches only — no restore-keys, since a near-miss would publish databases built from inputs the commit does not describe.
This commit is contained in:
1 parent
4cb0a4f340
commit
d8b5b66fcf
8 files changed
+337
-34
No files matched your search
@@ -96,12 +96,35 @@ jobs:
|
||||
GOVULNCHECK="$(go env GOPATH)/bin/govulncheck"
|
||||
for m in parser crawler assembler; do (cd "$m" && "$GOVULNCHECK" ./...); done
|
||||
|
||||
# One command runs the whole pipeline: compile the parser, build and
|
||||
# verify each database against its registry row count and size, build the
|
||||
# web app, and assemble _site — refusing to continue if a database is
|
||||
# short, an artifact looks truncated, or one is missing entirely.
|
||||
# Parsing 348 MB of spreadsheets is the slowest part of the job, and its
|
||||
# output is a pure function of the inputs in the key: the source files,
|
||||
# the parser (schema, transforms, configs) and the registry the row-count
|
||||
# and size guards read. Anything else — web code, docs, workflow — cannot
|
||||
# change a database, so those pushes restore one instead of rebuilding.
|
||||
#
|
||||
# Exact matches only, with no restore-keys: a near-miss here would mean
|
||||
# publishing a database built from different inputs than the commit says.
|
||||
- name: Cache the built databases
|
||||
id: db-cache
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: .build/public/db
|
||||
key: db-${{ hashFiles('data/**', 'parser/**', 'datasets.json') }}
|
||||
|
||||
# The whole pipeline: compile the parser, build and verify each database
|
||||
# against its registry row count and size, build the web app, and
|
||||
# assemble _site — refusing to continue if a database is short, an
|
||||
# artifact looks truncated, or one is missing entirely. On a cache hit
|
||||
# the database step is skipped; the artifact it would have produced was
|
||||
# verified by the run that built and stored it.
|
||||
- name: Build site
|
||||
run: go -C assembler run ./cmd/assemble
|
||||
run: |
|
||||
if [ "${{ steps.db-cache.outputs.cache-hit }}" = "true" ]; then
|
||||
echo "Databases restored from cache; building the site only."
|
||||
go -C assembler run ./cmd/assemble site
|
||||
else
|
||||
go -C assembler run ./cmd/assemble
|
||||
fi
|
||||
|
||||
- uses: actions/upload-pages-artifact@v5
|
||||
with:
|
||||
|
||||
Reference in new issue
Block a user