feat: split data update from site build; publish via Cloudflare Pages

The updater fetched GitHub metadata and rendered the dashboard payload in
a single step, so anything that published the site needed a GITHUB_TOKEN.
Cloudflare Pages builds from a Git webhook and has no business holding
one.

Split the tool into three modes. The default update step still fetches
GitHub and now records every API-sourced field in data/metadata.json,
which is committed alongside README.md and data/history.jsonl. The new
-build mode joins that snapshot with data/agents.yml and
data/history.jsonl to render dist/ with no network access and no token;
-check is unchanged.

Supporting changes:

- computeDeltaAt anchors the delta windows to when the data was fetched
  rather than the wall clock, so a redeploy days later reproduces the
  same Δ7d instead of sliding the window past its slack allowance.
- writeSiteData takes updatedAt explicitly for the same reason: the
  timestamp labels data freshness, not build time.
- sortStats is extracted from fetchStats so the build step re-ranks
  identically from committed metadata.
- An agents.yml entry with no metadata yet is omitted with a warning
  instead of failing the build, which would otherwise block every deploy
  between merging a new entry and the next nightly run.
- update.yml drops the GitHub Pages deploy steps and commits
  data/metadata.json; ci.yml runs `go run . -build` so a build that would
  break on deploy breaks in CI first.
- site/_headers stops the edge serving a stale data.json after a refresh.
- docs/DEPLOY.md covers the Cloudflare setup, including the one-time
  bootstrap of data/metadata.json that the build depends on.

Also carries the in-progress curation work already in the tree: the
module rename to awesome-ai-dev-tools, retagged entries, and removal of
the archived Roo-Code, void, continue and suna entries.
This commit is contained in:
tiennm99 committed 2026-09-16 22:24:48 +07:00
1 parent 3310ed36d4
commit 00c90c5be9
22 files changed
+728 -116

No files matched your search

+36 -4
View File
@@ -9,25 +9,56 @@
1. Go to [https://github.com/settings/tokens](https://github.com/settings/tokens)
2. Click "Generate new token" → "Generate new token (classic)"
3. Give it a name (e.g., "ai-dev-tools")
3. Give it a name (e.g., "awesome-ai-dev-tools")
4. Select scope: **`public_repo`** (needed to read public repo metadata)
5. Click "Generate token" and copy the value
The updater reads public repos only; `public_repo` scope is sufficient and safe.
## Running Locally
## The two steps
The tool separates fetching data from rendering the site, so only the fetch
needs a token. See [DEPLOY.md](./DEPLOY.md) for why.
### Update (needs a token, hits the network)
```bash
export GITHUB_TOKEN=ghp_your_token_here
go run .
```
The tool will:
This will:
- Read `data/agents.yml`
- Fetch live metadata from GitHub GraphQL API
- Fetch live metadata from the GitHub GraphQL API
- Append star counts to `data/history.jsonl`
- Write the fetched fields to `data/metadata.json`
- Regenerate `README.md` from `templates/readme.tmpl`
### Build (offline, no token)
```bash
go run . -build
```
This joins `data/agents.yml` (tags and notes) with `data/metadata.json` (stars
and repo metadata) and renders `dist/` — a copy of `site/` plus the generated
`dist/data.json`. This is exactly what Cloudflare Pages runs.
Preview it with any static server:
```bash
go run . -build && python3 -m http.server -d dist 8080
```
If you only touched `site/index.html` or the tags in `data/agents.yml`, the
build step alone is enough — no token required.
### Validate (offline, no token)
```bash
go run . -check
```
## Reverting Local Changes
Before opening a PR, undo local modifications:
@@ -48,3 +79,4 @@ Always set `GITHUB_TOKEN` when testing locally to avoid hitting the unauthentica
## Next Steps
To add agents, see [CONTRIBUTING.md](./CONTRIBUTING.md).
To publish the site, see [DEPLOY.md](./DEPLOY.md).