diff --git a/.claude/skills/mt-add-article/SKILL.md b/.claude/skills/mt-add-article/SKILL.md index b0f3a28..b250d7b 100644 --- a/.claude/skills/mt-add-article/SKILL.md +++ b/.claude/skills/mt-add-article/SKILL.md @@ -15,19 +15,20 @@ A clean article URL (passed by `mt-add-url`, or given directly). If a raw URL is ```bash node scripts/newsletter add-url "" ``` -Trust `route: article`; skip if `duplicate` or not `accessible`. +Trust `route: article`; skip if `duplicate`; for `accessible: false` follow post-mechanics §6 (not a skip by itself). ## Workflow -1. **Post mechanics** — follow `docs/newsletter/post-mechanics.md` to resolve/create the target post and (if new) get the newsletter number + apply the template. -2. **Extract** the original article title and main content. Try `WebFetch` first; if the host blocks it (403, Cloudflare challenge, empty body), work through the `mt-fetch-url` fallback chain before giving up. A `accessible: false` from the router is not by itself a reason to skip — many public tech blogs block plain fetches but serve the fallback fetchers fine. Preserve the source title exactly enough to remain recognizable: do not translate/localize it; keep source-language wording, capitalization, punctuation, and proper nouns from metadata (`og:title`, page title, or fetcher frontmatter). -3. **Summarize in Vietnamese** — 1-2 paragraphs, max 300 words, professional tone for junior developers. Summary paragraphs only; do not add a key-points bullet list. +1. **Extract** the original article title and main content. Try `WebFetch` first, then the `mt-fetch-url` fallback chain. Title: post-mechanics §4 (*Title source*). Skip rules: post-mechanics §6. + - **GitHub repo URLs** (`github.com//`): `WebFetch` returns only the repo chrome with HTTP 200, so the fallback never triggers. Read the README from `https://raw.githubusercontent.com///HEAD/README.md` (`HEAD` resolves the default branch). On 404, find the README's real name in the repo page's file list. The title follows the post-mechanics §4 GitHub rule. +2. **Summarize in Vietnamese** — length per post-mechanics §5 *Summary length* (count with `awk` NF, target 280–298, max 300 words); professional tone for junior developers. Summary paragraphs only; do not add a key-points bullet list. +3. **Post mechanics** — follow `docs/newsletter/post-mechanics.md` to resolve/create the target post and (if new) get the newsletter number + apply the template. Done after extraction so a failed fetch never leaves an empty new newsletter behind. 4. **Write** the main-content block and insert it **before** the `### Bonus` section (or append at end of file if there is no Bonus yet — do not create an empty Bonus): ```markdown ## [Original article title](clean_url) -[Tóm tắt tiếng Việt — tối đa 300 từ] +[Tóm tắt tiếng Việt — ≤300 từ, xem post-mechanics §5] ``` 5. **Report**: @@ -37,7 +38,7 @@ Trust `route: article`; skip if `duplicate` or not `accessible`. 🔗 [clean_url] 📊 [tally] ``` -The tally comes from `post-stats` — see *Post tally* in the shared post mechanics. +The `📊` line uses the post-mechanics §4a (*Post tally*) format. ## Checklist @@ -47,5 +48,6 @@ The tally comes from `post-stats` — see *Post tally* in the shared post mechan - [ ] URL is clean (no tracking params) - [ ] Heading uses the original source article title, not a Vietnamese translation - [ ] Summary paragraphs only, no key-points bullet list +- [ ] Summary ≤300 words by awk NF count (target 280-298), per post-mechanics §5 - [ ] Vietnamese content ≥99%, grammar correct - [ ] Article inserted before the Bonus section diff --git a/.claude/skills/mt-add-image/SKILL.md b/.claude/skills/mt-add-image/SKILL.md index fc4ffad..9cbfba1 100644 --- a/.claude/skills/mt-add-image/SKILL.md +++ b/.claude/skills/mt-add-image/SKILL.md @@ -11,7 +11,7 @@ Shared scripts: `scripts/newsletter/`. Shared procedure: `docs/newsletter/post-m **Label priority:** figure caption → source post title → your typed input. -**Label language:** Preserve the selected label's original source-language wording, capitalization, and punctuation. Do not translate or localize it unless the user explicitly asks. Preserve user-typed labels exactly as entered. +**Label language:** keep the source language per `AGENTS.md` / post-mechanics §5; preserve user-typed labels exactly as entered. ## Input @@ -27,7 +27,7 @@ node scripts/newsletter detect-image-source "" When invoked **directly** (not via `mt-add-url`), first run the router to get accessibility + duplicate status and skip accordingly: ```bash -node scripts/newsletter add-url "" # expect route:image; skip if duplicate/!accessible +node scripts/newsletter add-url "" # expect route:image; skip if duplicate (report where, per post-mechanics §6); for `accessible: false` follow post-mechanics §6 ``` (When dispatched by `mt-add-url`, that check already ran — don't repeat it.) @@ -43,17 +43,19 @@ node scripts/newsletter find-substack-post --uuid ```bash node scripts/newsletter find-substack-post --uuid --deep ``` - On a miss the result reports `scanned` (posts fetched), `budget` (the 40-fetch cap), and `cutoff` (oldest date looked at) — mention how far back it looked. + On a miss the result reports `scanned` (posts fetched), `budget` (the 40-fetch cap), and `cutoff`. `cutoff` is the fixed 3-month boundary (`null` means no sitemap could be fetched). If `scanned` equals `budget`, the budget ran out before the cutoff, so say older posts were not checked. - `found: false` after `--deep` → no source post; go to step 3 (ask) and/or step 4 (add publication). **When `found: true` — pick the label (confirm-from-candidates):** ByteByteGo doesn't attach captions to images (verified: empty `alt`, no `
`), and image order can't be mapped to titles automatically because sponsor/video items interleave. So the script returns `candidates` — the newsletter post's topic titles from the post's TOC. Pick the label this way: 1. If `caption` is non-empty (some publications do caption images) → propose it as the default. -2. Otherwise present `candidates` to the user via `AskUserQuestion` (the correct title is in this list): +2. Otherwise, **recommend, don't guess.** Fetch `/api/v1/posts/` (slug from `postUrl`) and walk `body_html` in document order, printing each `

`–`

` text and each deduped `images/`. The nearest heading above this image's UUID is the recommended label. Reuse the same ordering for later images from the same post. +3. Present `candidates` via `AskUserQuestion`. Candidates = `
  • ` texts of 6–70 chars from the post (usually its TOC); the right title is usually, but not always, among them: - "Image is from **[postTitle]** (`postUrl`). Which title matches it?" + - List the recommended title first, marked (Recommended), and cite the evidence (the heading above the UUID); the user still confirms. If no candidate fits, offer that heading as an option. - Offer the candidates. If there are more than 4, show them as a numbered list in text and ask the user to pick a number (AskUserQuestion allows max 4 options). - Always include a "Type my own" escape. -3. Label = the user's pick (or the typed value). Preserve its original language, wording, capitalization, and punctuation; do not translate unless the user explicitly asks. +4. Label = the user's pick (or the typed value), in its source language (see **Label language** above). **Presenting candidates — rules that prevent mis-labelling:** - **Dedupe first.** The list can contain the same title twice, once with literal escape sequences (e.g. `\u2019` instead of `'`). Show each title once, in its readable form. diff --git a/.claude/skills/mt-add-tags/SKILL.md b/.claude/skills/mt-add-tags/SKILL.md index a42e949..9c8716a 100644 --- a/.claude/skills/mt-add-tags/SKILL.md +++ b/.claude/skills/mt-add-tags/SKILL.md @@ -100,20 +100,16 @@ Replace the entire existing `tags:` line. Preserve all other frontmatter fields ## Proactive behavior (before commit) -When the user is about to commit (`/commit`, `mt:commit-push-pr`, or similar), find the most recently modified `content/post/*/index.md` file: +Before any git commit (the AGENTS.md pre-commit rule), list every new or modified post, staged or not: ```bash -git diff --name-only HEAD | grep "^content/post/" | head -1 +git status --porcelain -uall -- 'content/post/*/index.md' ``` -If that returns nothing, check unstaged changes: +Git's `*` matches across `/`, so this covers untracked new posts and modified ones. Check every listed post, not just one, so a batch of backfilled newsletters is covered. -```bash -git status --short | grep "content/post/" | awk '{print $2}' | head -1 -``` - -If a post file is found and its tags look minimal (only 1-2 tags, or only `AI-Assisted`/`Newsletter`), say: +A post's tags are **minimal** when they are empty, only `AI-Assisted` and/or `Newsletter`, or below target (2-3 for regular posts, 6-7 for newsletters, `AI-Assisted` included). For each minimal post, say: > "Post `content/post/.../index.md` only has these tags: [...]. Add richer tags before committing? [Y/n]" -Default is **yes** — proceed to generate and confirm tags unless the user declines. +Default is **yes** — proceed to generate tags, then confirm them with AskUserQuestion before writing (step 5), unless the user declines. diff --git a/.claude/skills/mt-add-url/SKILL.md b/.claude/skills/mt-add-url/SKILL.md index a696eda..39ad087 100644 --- a/.claude/skills/mt-add-url/SKILL.md +++ b/.claude/skills/mt-add-url/SKILL.md @@ -29,21 +29,21 @@ Output (JSON): `{ original_url, clean_url, http_status, accessible, duplicate, r ### 2. Skip non-actionable URLs -- `duplicate: true` → skip, note in report (already in a newsletter). -- `accessible: false` → **not an automatic skip.** The classifier does a plain fetch, so a bot-blocked host (403, Cloudflare challenge) reports `accessible: false` even when the page is public and the fallback fetchers can read it. Dispatch on `route` as normal and let the handler's fetch chain decide; only report the URL as skipped when every fetcher in `mt-fetch-url` has failed. A `404`/dead URL is a genuine skip. +- `duplicate: true` → skip and report where it already is, per `docs/newsletter/post-mechanics.md` §6. +- `accessible: false` → **not an automatic skip** (HEAD probe; bot-blocked hosts read false). Dispatch on `route` as normal and follow post-mechanics §6. ### 3. Dispatch on route | route | Action | |-------|--------| | `article` | Invoke the **`mt-add-article`** skill, passing `clean_url` | -| `youtube` | Invoke the **`mt-add-video`** skill, passing `clean_url` | +| `youtube` | Invoke the **`mt-add-video`** skill, passing `clean_url`, `title` and `author` | | `image` | Invoke the **`mt-add-image`** skill, passing `clean_url` | | `video` (direct file) / `document` / anything else | **Fallback** — see step 4 | Dispatch by calling the Skill tool for the chosen handler with `clean_url` as the argument. The handler completes the write end-to-end. -**Multiple URLs:** dispatch **sequentially** (one handler finishes before the next starts) — handlers edit the same daily `index.md`, so concurrent edits would clobber each other. Process article(s) and video(s) one at a time. +**Multiple URLs:** dispatch **sequentially** (one handler finishes before the next starts) — handlers edit the same daily `index.md`, so concurrent edits would clobber each other. Process every URL (articles, videos, images) one at a time. ### 4. Fallback for unsupported types @@ -69,8 +69,7 @@ Aggregate across all URLs: ``` ✅ Newsletter URL Dispatch Complete -📊 Newsletter #[number]: [articles] articles · [videos] videos · [images] images - (omit zero counts; documents too when present) +📊 [tally line in the post-mechanics §4a format] ✅ Dispatched: [count] - [count] → mt-add-article (articles) diff --git a/.claude/skills/mt-add-video/SKILL.md b/.claude/skills/mt-add-video/SKILL.md index fb93905..e00ab63 100644 --- a/.claude/skills/mt-add-video/SKILL.md +++ b/.claude/skills/mt-add-video/SKILL.md @@ -17,20 +17,25 @@ A clean YouTube URL (passed by `mt-add-url`, or given directly). ## Workflow -1. **Classify / fetch title** — run the router to get the canonical URL + title: +1. **Classify / fetch title** — when dispatched by `mt-add-url`, reuse its `clean_url`, `title` and `author` (channel) and do not rerun the router. Only when invoked directly, run it to get the canonical URL, title and channel: ```bash node scripts/newsletter add-url "" ``` - Confirm `route: youtube`; skip if `duplicate` or not `accessible`. Use the returned `clean_url` (canonical `watch?v=ID`) and `title`. - - If `title` is missing (oEmbed failed), fetch the title via WebFetch on the watch URL. + Confirm `route: youtube`; skip if `duplicate`; for `accessible: false` follow `docs/newsletter/post-mechanics.md` §6. Use `clean_url` (canonical `watch?v=ID`), `title` and `author` (channel). + - If `title` is missing (oEmbed failed), take the `videoDetails` `"title"` from the same watch-page curl as step 3, not WebFetch. -2. **Localize title** — render the title in Vietnamese where natural; keep proper nouns (product/company/channel names) as-is. +2. **Localize title** (intentional exception to the keep-source-language rule for article titles and image labels — see AGENTS.md) — keep the original title when it is mostly proper nouns, product names or a code identifier; otherwise translate it into Vietnamese and keep the proper nouns (product/company/channel names) as-is. 3. **Write a 1-2 sentence Vietnamese summary.** Source for the summary content, in order: - - WebFetch the watch URL for the video description. - - If thin/unavailable, optionally use the `ai-multimodal` (Gemini) skill to analyze the video — only if available. - - Worst case, derive a single sentence from the title. - Keep it to 1-2 sentences (KISS). + 1. The watch page's player JSON (WebFetch usually returns no description): + ```bash + curl -sL -A 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124 Safari/537.36' -H 'Accept-Language: en' '' | grep -oE '"shortDescription":"([^"\\]|\\.)*"' | head -1 + ``` + Then JSON-unescape the value (`\n`, `\"`). + 2. WebFetch on the watch URL. + 3. An installed multimodal skill (e.g. `ak:ai-multimodal`), only if available. + 4. As a last resort, derive the summary from title + channel and say so in the report. + Blockquote length per post-mechanics §5. 4. **Post mechanics** — follow `docs/newsletter/post-mechanics.md` to resolve/create the target post and locate the Bonus section. @@ -55,7 +60,7 @@ A clean YouTube URL (passed by `mt-add-url`, or given directly). ## Checklist - [ ] `route: youtube` confirmed; not a duplicate -- [ ] Title in Vietnamese (proper nouns kept) +- [ ] Title localized per step 2 (Vietnamese with proper nouns kept, or original when it is mostly proper nouns) - [ ] 1-2 sentence Vietnamese summary present - [ ] Entry under Bonus → **Videos** (section/subsection created if missing) - [ ] Vietnamese content ≥99% diff --git a/.claude/skills/mt-fetch-url/SKILL.md b/.claude/skills/mt-fetch-url/SKILL.md index 30418cb..a8a85ae 100644 --- a/.claude/skills/mt-fetch-url/SKILL.md +++ b/.claude/skills/mt-fetch-url/SKILL.md @@ -1,6 +1,6 @@ --- name: mt-fetch-url -description: "Fallback web content fetchers for pages that blocked the built-in fetch. Use ONLY when the built-in WebFetch tool has already failed with 403 Forbidden, bot detection, Cloudflare challenge, empty content, or similarly blocked response. Tries local defuddle extraction, then the defuddle.md proxy, then a reader proxy — the proxies fetch the page server-side from a different IP and return clean markdown. Do NOT use as a first-choice fetcher — try WebFetch first. Does NOT bypass paywalls, login walls, or pages that require JavaScript execution." +description: "Fallback web content fetchers for pages that blocked the built-in fetch. Use ONLY when the built-in WebFetch tool has already failed with 403 Forbidden, bot detection, Cloudflare challenge, empty content, or similarly blocked response. Tries local defuddle extraction, then the defuddle.md proxy, then a reader proxy — the proxies fetch the page server-side from a different IP and return clean markdown — then renders the page in a local headless browser (agent-browser) for JavaScript-only pages. Do NOT use as a first-choice fetcher — try WebFetch first. Does NOT bypass paywalls, login walls, or Cloudflare challenge pages that also stop a headless browser." --- ## Scope @@ -9,7 +9,7 @@ This skill handles: fetching public web pages that blocked WebFetch due to bot-d This skill does NOT handle: - Paywalled or login-gated content -- Pages requiring client-side JavaScript execution (these proxies do HTTP fetch, not headless rendering) +- Cloudflare "Just a moment…" challenges that also stop headless Chromium (tested: the agent-browser tier stays on the challenge page) - URLs that return 404 / are actually dead - Sites that block every fetcher in the chain below @@ -39,10 +39,24 @@ Use this skill only after a WebFetch attempt returned one of: WebFetch(url: "https://r.jina.ai/", prompt: "") ``` Tier 1 and tier 2 fail independently — a site blocking one often still serves the other, so always attempt tier 2 before giving up. -4. Parse the returned markdown — both tier-1 stages emit YAML frontmatter with title/description/etc. ahead of the body -5. If every tier fails, stop and report which tiers were tried and what each returned — do not keep retrying. +4. **Tier 3 — local headless browser (agent-browser).** If tier 2 also fails or returns only SPA boilerplate, render the page in a real browser and read the rendered DOM: + ```bash + export AGENT_BROWSER_SESSION=mt-fetch + npx -y agent-browser open "" && + npx -y agent-browser wait 10000 && + npx -y agent-browser get title && + npx -y agent-browser read + npx -y agent-browser close + ``` + - Run the whole block as ONE shell invocation: environment variables do not persist between agent shell calls, so a later call would lose `AGENT_BROWSER_SESSION` and talk to a different session. The `&&` chain stops at the first failure, and `close` runs regardless. + - Where agent-browser cannot download its own Chrome (no Chrome for Testing build, e.g. Linux ARM64), also export `AGENT_BROWSER_EXECUTABLE_PATH` to the system Chromium in that same invocation. + - `read` with no URL reads the rendered DOM of the open tab. Its output is the page text with menus and banners kept — skip that noise when summarizing; no extra extraction step is needed. + - Always run `close` at the end, even after a failure, so no headless browser is left running. + - If the title is still "Just a moment…" or the body is a verification message, the page is behind a bot challenge: count the tier as failed. +5. Parse the returned markdown — both tier-1 stages emit YAML frontmatter with title/description/etc. ahead of the body; tier 3 gives the title via `get title` +6. If every tier fails, stop and report which tiers were tried and what each returned — do not keep retrying. -Attempt each tier at most once. The whole chain is: built-in WebFetch → local defuddle → defuddle.md → reader proxy → report failure. +Attempt each tier at most once. The whole chain is: built-in WebFetch → local defuddle → defuddle.md → reader proxy → agent-browser → report failure. ## How defuddle works @@ -95,5 +109,8 @@ WebFetch returned: "Request failed with status code 403" → exit 1 after both stages: continue → Tier 2: WebFetch(url: "https://r.jina.ai/https://example.com/article", prompt: ...) → success: parse markdown output, summarize as usual - → failure: report "WebFetch 403, defuddle 502, reader proxy failed" and stop + → failure: continue +→ Tier 3: agent-browser open → wait → get title → read → close + → success: summarize the rendered text as usual + → challenge page or empty: report "WebFetch 403, defuddle 502, reader proxy failed, agent-browser stuck on challenge" and stop ``` diff --git a/.claude/skills/mt-rewrite-newsletter/SKILL.md b/.claude/skills/mt-rewrite-newsletter/SKILL.md index acf11da..535debb 100644 --- a/.claude/skills/mt-rewrite-newsletter/SKILL.md +++ b/.claude/skills/mt-rewrite-newsletter/SKILL.md @@ -56,9 +56,9 @@ Anything else that is not an AI summary paragraph — a blank-line separator, a Keep the `post-stats` counts for step d. b. **Rewrite each entry summary** (headings unchanged): - - Re-read the source: `WebFetch` the entry URL; if blocked, use the `mt-fetch-url` chain. YouTube entries: use oEmbed title/description plus the existing summary. + - Re-read the source: `WebFetch` the entry URL; if blocked, use the `mt-fetch-url` chain. YouTube entries: title/author from `node scripts/newsletter add-url ` (oEmbed has no description) plus the existing summary; description source per `mt-add-video` step 3. - Source unreachable → rewrite from the existing summary only; add no new facts. Note it in the report. - - Output per post-mechanics rules: Vietnamese (≥99%), 1–2 prose paragraphs, ≤300 words, no key-points bullet list, junior-developer audience, professional tone. Video blockquote summaries (`> …`) stay 1–2 sentences. + - Output per post-mechanics rules: Vietnamese (≥99%), length per post-mechanics §5 (Summary length), no key-points bullet list, junior-developer audience, professional tone. - Struck entries (`## ~~[…]~~`): leave the heading and every body line untouched. - Edit summary paragraphs in place with anchored `Edit` calls; never rewrite the whole file. @@ -94,7 +94,7 @@ Give each subagent: the post path, model + tool name, today's date string, the s - [ ] `protected-lines --against` → `ok: true` - [ ] `post-stats` counts unchanged - [ ] Exactly one provenance note, last line, correct model + DD/MM/YYYY date -- [ ] Summaries Vietnamese, prose, ≤300 words; headings and image labels untouched +- [ ] Summaries Vietnamese, prose, ≤300 words by awk NF count (post-mechanics §5); headings and image labels untouched - [ ] Struck entries and author-voice lines byte-identical ## Security diff --git a/AGENTS.md b/AGENTS.md index 2457818..ce59e25 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,7 +10,7 @@ Canonical instructions for AI coding tools (Claude Code, OpenCode, Codex) workin - **User communication**: English by default; use another language only when the user explicitly requests it - **Timezone**: Asia/Ho_Chi_Minh (UTC+7) -Vietnamese language requirements apply only to prose written into Hugo posts, including summaries. Preserve article titles and image labels in their original source language, wording, capitalization, and punctuation; translate them only when the user explicitly asks. Keep questions, status updates, reports, and final responses in English unless the user asks for another language. +Vietnamese language requirements apply only to prose written into Hugo posts, including summaries. Preserve article titles and image labels in their original source language, wording, capitalization, and punctuation; translate them only when the user explicitly asks. Keep questions, status updates, reports, and final responses in English unless the user asks for another language. YouTube video titles in Bonus → Videos are the deliberate exception: they are localized into Vietnamese, keeping proper nouns and product names (see `mt-add-video`). ## Directory Structure @@ -50,7 +50,8 @@ The eight subcommands, their arguments, output shapes and exit codes are documented once in **[docs/newsletter/engine-commands.md](docs/newsletter/engine-commands.md)**. The engine has npm dependencies, so a fresh clone needs `npm ci` from the repo -root once before any skill will run. +root once; Substack image lookup and local defuddle extraction depend on it +(details in engine-commands.md). The engine is shared by all three tools — no tool-specific copies. @@ -67,13 +68,15 @@ The newsletter workflow adds URLs (articles, YouTube videos, images) to the targ - `mt-add-image` — image → Bonus → Images (labels Substack images via source-post lookup) - `mt-add-tags` — add/update tags in post frontmatter - `mt-rewrite-newsletter` — rewrite existing newsletter summaries with a newer model, keeping handwritten lines and stamping a rewrite note - - `mt-fetch-url` — fallback web fetch chain (local defuddle, the defuddle.md proxy, then a reader proxy); use only when built-in WebFetch is blocked + - `mt-fetch-url` — fallback web fetch chain (local defuddle, the defuddle.md proxy, a reader proxy, then a local headless browser via agent-browser); use only when built-in WebFetch is blocked - **Codex** — discovers the repository-scoped adapters in `.agents/skills/`. Ask it to add a URL for implicit routing or invoke `$mt-add-url` explicitly. `mt-add-url` dispatches `article` / `youtube` / `image`; other types (direct video files, documents, unknown) prompt the user to add or extend a handler. Canonical skill implementations live in `.claude/skills/`; `.agents/skills/` symlinks to them so there is only ever one copy to edit. +Skills name Claude Code tools; other tools map them as follows. `WebFetch` → the tool's web fetch, or `curl -sL`. `AskUserQuestion` → the tool's question prompt, or a numbered list in chat, then wait. Skill tool → open and follow `.agents/skills//SKILL.md`. `Edit` → an anchored edit, never a whole-file rewrite. `` → `temp/`. Subagents → process posts sequentially. + ### Using Codex Codex automatically discovers checked-in skills from `.agents/skills/`. No install or copy step is required. Start Codex at the repository root, then either describe the task normally or explicitly mention a skill: @@ -92,4 +95,6 @@ Codex detects skill changes automatically; restart Codex if an update does not a ## Git Workflow Rules -**Before any git commit**, check all staged `content/post/*/index.md` files for minimal tags. If any have only `["AI-Assisted"]` or empty tags, run the `mt-add-tags` workflow on them first before committing. +**Before any git commit**, run the `mt-add-tags` pre-commit check on every staged or new `content/post/YYYY/MM/DD/index.md` (the skill defines what counts as minimal tags). + +Stage only the post files you changed, because unrelated local edits may be present; never `git add -A`. Commit as `docs(newsletter): add newsletter N`. A daily bot commit (`chore(projects): refresh projects list`, from `.github/workflows/update-projects.yml`) often lands on main first, so push with `git pull --rebase --autostash && git push`. diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..09aa505 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,21 @@ +# Roadmap + +Ideas not yet scheduled. Move an item into a plan once it is decided. + +## Cloudflare Browser Run as a fetch tier for `mt-fetch-url` + +Cloudflare's Browser Run (formerly Browser Rendering) `/markdown` quick action renders a URL in Cloudflare's headless browser and returns Markdown: + +```bash +curl -X POST "https://api.cloudflare.com/client/v4/accounts/$CF_ACCOUNT_ID/browser-run/markdown" \ + -H "Authorization: Bearer $CF_BROWSER_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"url":"https://example.com","gotoOptions":{"waitUntil":"networkidle2"}}' +# -> {"success": true, "result": ""} +``` + +- **Why:** it fetches from a different exit IP than this machine, which local agent-browser cannot. +- **Limits:** requests are signed as a bot (Web Bot Auth), so it is not expected to pass bot challenges. The free plan allows 10 browser-minutes a day and 6 REST requests a minute. +- **Blocked on:** a Cloudflare account ID and an API token with only the Browser Run – Edit permission, kept in environment variables and never committed. +- **Done when:** the token is configured, testing shows it recovers pages the current chain (WebFetch → defuddle → defuddle.md → reader proxy → agent-browser) misses, and it is added to `.claude/skills/mt-fetch-url/SKILL.md` as the last tier. +- **Docs:** [Browser Run /markdown](https://developers.cloudflare.com/browser-run/quick-actions/markdown-endpoint/), [limits](https://developers.cloudflare.com/browser-rendering/limits/). diff --git a/docs/multi-tool-usage.md b/docs/multi-tool-usage.md index 52f3814..883f1b3 100644 --- a/docs/multi-tool-usage.md +++ b/docs/multi-tool-usage.md @@ -8,22 +8,22 @@ This repo is usable from three AI coding tools off **one shared engine**. The ne |------|------------------|---------------------------|-------|--------------------| | **Claude Code** | `CLAUDE.md` → imports `AGENTS.md` | Native skill auto-dispatch from `.claude/skills/` | `npm ci` once | Paste a URL; `mt-add-url` auto-runs | | **OpenCode** | `AGENTS.md` (auto-read) | `.claude/skills/` auto-discovered via the `skill` tool, governed by `opencode.json` | `npm ci` once; `opencode.json` is committed | Ask to add a URL; pick/`skill` `mt-add-url` | -| **Codex** | `AGENTS.md` (auto-read) | Repository skills discovered from `.agents/skills/` | `npm ci` once | Ask to add a URL or invoke `$mt-add-url` | +| **Codex** | `AGENTS.md` (auto-read) | Repository skills discovered from `.agents/skills/` | `npm ci` once; network access enabled | Ask to add a URL or invoke `$mt-add-url` | **Shared engine:** all three call `node scripts/newsletter ` from repo root — no per-tool script copies. It has npm dependencies, so run `npm ci` once after cloning. ### Notes per tool -- **Claude Code** — `CLAUDE.md` is a thin file that imports `AGENTS.md` (`@AGENTS.md`); all instruction content lives in `AGENTS.md`. Permissions in `.claude/settings.json`. -- **OpenCode** — reads `.claude/skills//SKILL.md` natively (multi-location search up the worktree), so the same `mt-*` skills work with zero duplication. `opencode.json` mirrors the Claude allow-list intent (node/git/hugo/edit). No MCP block — relies on OpenCode's built-in web fetch/search. Restart OpenCode after editing `opencode.json` (config is not hot-reloaded). -- **Codex** — reads repository skills from `.agents/skills/` and can invoke them explicitly (`$mt-add-url`) or implicitly from their descriptions. Each `.agents/skills//SKILL.md` is a **symlink** to its `.claude/skills/` counterpart, so there is one file to edit and nothing that can drift. If a checkout cannot follow symlinks (Windows without developer mode), replace them with thin pointer files that say "read and follow `.claude/skills//SKILL.md`". +- **Claude Code** — `CLAUDE.md` is a thin file that imports `AGENTS.md` (`@AGENTS.md`); all instruction content lives in `AGENTS.md`. No Claude permission file is committed; per-user permissions go in the gitignored `.claude/settings.local.json`. +- **OpenCode** — reads `.claude/skills//SKILL.md` natively (multi-location search up the worktree), so the same `mt-*` skills work with zero duplication. `opencode.json` is the only committed permission policy: it allows the engine (`node scripts/newsletter`), read-only and commit git commands, `hugo`, `curl`, agent-browser and basic file/text utilities; everything else asks. Rules are last-match-wins, so keep `"*": "ask"` first. No MCP block — relies on OpenCode's built-in web fetch/search. Restart OpenCode after editing `opencode.json` (config is not hot-reloaded). +- **Codex** — reads repository skills from `.agents/skills/` and can invoke them explicitly (`$mt-add-url`) or implicitly from their descriptions. Each `.agents/skills//SKILL.md` is a **symlink** to its `.claude/skills/` counterpart, so there is one file to edit and nothing that can drift. If a checkout cannot follow symlinks (Windows without developer mode), replace them with thin pointer files that say "read and follow `.claude/skills//SKILL.md`". The engine and the fetch tiers need outbound network: run Codex with network enabled (e.g. `sandbox_workspace_write.network_access = true` in your Codex config), or approve the escalation when prompted. ## Teardown — pick one, remove the rest -Once you commit to one tool, delete the others' config: +Once you commit to one tool, delete the others' config. There are no committed Claude settings; `.claude/settings.local.json` is personal and gitignored. - **Keep Claude Code only** → remove `opencode.json` and `.agents/`. Keep `.claude/` and `docs/newsletter/`. -- **Keep OpenCode only** → remove `.claude/settings*.json` and `.agents/`. **Keep `.claude/skills/`** — OpenCode reads it. If you want to drop the `.claude/` dir entirely, first move skills to `.opencode/skills/` (scripts are path-neutral in `scripts/newsletter/`, so the move breaks nothing). Optionally make `AGENTS.md` the only instruction file (delete `CLAUDE.md`). -- **Keep Codex only** → keep `.agents/skills/`, `AGENTS.md`, `scripts/newsletter/`, `docs/newsletter/`, and the canonical `.claude/skills/` files the Codex skills symlink to. You may remove Claude-specific settings and `opencode.json`. +- **Keep OpenCode only** → remove `.agents/`. **Keep `.claude/skills/`** — OpenCode reads it. If you want to drop the `.claude/` dir entirely, first move skills to `.opencode/skills/` (scripts are path-neutral in `scripts/newsletter/`, so the move breaks nothing). Optionally make `AGENTS.md` the only instruction file (delete `CLAUDE.md`). +- **Keep Codex only** → keep `.agents/skills/`, `AGENTS.md`, `scripts/newsletter/`, `docs/newsletter/`, and the canonical `.claude/skills/` files the Codex skills symlink to. You may remove `opencode.json`. In every case: **keep `scripts/newsletter/`, `.claude/skills/`, `docs/newsletter/`, and `AGENTS.md`** — they are the shared engine, canonical workflows, shared documentation, and instructions. diff --git a/docs/newsletter/engine-commands.md b/docs/newsletter/engine-commands.md index b6f8ee7..a29389e 100644 --- a/docs/newsletter/engine-commands.md +++ b/docs/newsletter/engine-commands.md @@ -13,11 +13,11 @@ a fresh clone needs one setup step: npm ci ``` -Skip it and the content-parsing subcommands print +Skip it and `find-substack-post` prints `newsletter engine: dependencies missing — run 'npm ci' from the repo root` -and exit 1 rather than a module-resolution stack trace. `fetch-via-defuddle` is -the exception: its local tier needs the dependencies, so without them it falls -through to the `defuddle.md` proxy and still succeeds. +and exits 1 rather than a module-resolution stack trace. `fetch-via-defuddle` +loses only its local stage, so it falls through to the `defuddle.md` proxy and +can still succeed. The other subcommands need no dependencies. ## Invocation @@ -32,6 +32,11 @@ JSON output is 2-space indented with a trailing newline, and is not HTML-escaped (URLs keep their `&` readable). Optional fields are **omitted** when empty rather than emitted as `""`. +Exit codes: 0 is success; 1 is a usage error, unknown command, missing +dependencies or uncaught error. `find-substack-post` exits 2 on a bad flag; +`fetch-via-defuddle` and `protected-lines --against` use the codes in their own +sections. + ## Commands ### `add-url ` @@ -59,7 +64,15 @@ Classifies and deduplicates one URL — the single entry point per URL. route, and absent when oEmbed fails. - `duplicate` is boundary-aware: a URL that is merely a *prefix* of one already stored is not a duplicate. Substack images dedupe on their image uuid, so - transform and size variants collapse onto one identity. + transform and size variants collapse onto one identity. It is a boolean and + does not name the post holding the URL; see + [post-mechanics §6](post-mechanics.md#6-error-handling) for finding and + reporting it. +- `http_status` is the final status (after redirects) of a HEAD request sent + with Node's default User-Agent, not a browser one — `"000"` on network error + or 10 s timeout. `accessible` is `http_status == "200"`, so a bot-blocking or + HEAD-rejecting host reads `false` (see + [post-mechanics §6](post-mechanics.md#6-error-handling)). ### `find-newsletter-number` @@ -121,19 +134,19 @@ fetched at all. ### `fetch-via-defuddle ` -Fallback fetcher for pages that blocked the built-in fetch. Two tiers, in order: +Fallback fetcher for pages that blocked the built-in fetch. Two stages, in order: local defuddle extraction, then the `defuddle.md` proxy (which fetches from its -own IP — the tier that matters when this machine's IP is the blocked one). +own IP — the stage that matters when this machine's IP is the blocked one). -Both tiers write YAML frontmatter followed by the markdown body, so the caller -parses one shape either way; stderr names which tier failed and why. An +Both stages write YAML frontmatter followed by the markdown body, so the caller +parses one shape either way; stderr names which stage failed and why. An extraction that looks like a bot challenge rather than the requested page counts -as a local-tier failure, so the proxy still gets its turn. +as a local-stage failure, so the proxy still gets its turn. | Exit | Meaning | |---|---| | 0 | content returned | -| 1 | every tier failed | +| 1 | both stages failed | | 2 | bad arguments | ### `post-stats ` @@ -157,8 +170,9 @@ Counts what a post already holds, so a handler can report a running tally. ### `protected-lines [--against ]` Lists the lines a rewrite (`mt-rewrite-newsletter`) must keep byte-for-byte: -frontmatter, headings (except `###` sub-headings inside a summary), `` -blocks, whole-line italic notes, `~~struck~~` lines, fenced code blocks, and +frontmatter, headings (except `###` sub-headings inside a summary), HTML blocks +opened by ``/``/`
    `/`

    `/`

    `, `**Đánh giá:** *…*` author +notes (kind `author-note`), whole-line italic notes, `~~struck~~` lines, fenced code blocks, and asset links / Bonus subsection labels. Paragraphs mentioning `mình` or `MiTi` come back as `candidates` for a human-style judgement; the machine provenance note (`*Bài viết đã được … bởi …*`) comes back as `note` and is not diff --git a/docs/newsletter/post-mechanics.md b/docs/newsletter/post-mechanics.md index 7fd3ab7..ebdd887 100644 --- a/docs/newsletter/post-mechanics.md +++ b/docs/newsletter/post-mechanics.md @@ -16,6 +16,7 @@ The target defaults to today's date but the user can pin a different one. **Target date resolution:** - Default: current date in `YYYY-MM-DD` (UTC+7). - **Pinned target:** when the user names a post or date to keep working on — typically an unpublished draft started on an earlier day ("keep adding to the 7/9 post") — that post is the target for the rest of the session, including across a date rollover mid-session. Do not silently start a new post for the new day; if a pin might have lapsed, ask before creating one. +- **Backfill:** when the user names a past date for a **new** newsletter ("the day after #134"), that date is the target. Create `content/post///
    /index.md` with `date: ` — never today's date — and treat it as pinned for the rest of the session, exactly like a pinned draft. Commits still happen at real time. Check `content/post/YYYY/MM/DD/index.md` for the resolved target date: - **Exists** → update this file. @@ -28,7 +29,7 @@ The newsletter number always comes from the target post, not from today's date. ```bash node scripts/newsletter find-newsletter-number ``` -Searches backwards from today for the most recent newsletter and returns the next number. Only needed when **creating** a new post — when the target post already exists, read its number from its own `title`. +Returns 1 + the highest `Newsletter #N` in the newest year directory that has one. It ignores today's date and the target date, and future-dated posts count (details: [engine-commands.md](engine-commands.md)). When backfilling a date earlier than an existing newsletter, check that the number fits between the neighbouring posts, and ask the user if it does not. Create backfilled posts in date order so numbers stay chronological. Only needed when **creating** a new post — when the target post already exists, read its number from its own `title`. ## 2a. Terminology @@ -53,7 +54,15 @@ categories: ["Newsletter"] **Never rewrite the whole `index.md`.** Always insert by anchoring an Edit on an existing string (e.g. `### Bonus`, `**Videos:**`) and prepending/appending around it. Multiple URLs targeting the same day's post must be applied **sequentially** so one edit doesn't clobber another. **Articles go before the `### Bonus` section; Bonus assets go inside it.** -Article headings must use the original source title. Do not translate/localize article titles; keep the source-language wording, capitalization, punctuation, and proper nouns from metadata (`og:title`, page title, or fetcher frontmatter). Summaries stay Vietnamese and are prose paragraphs only — no key-points bullet list. +Article headings must use the original source title. Do not translate/localize article titles; keep the source-language wording, capitalization, punctuation, and proper nouns from metadata. + +**Title source:** +1. Use the first that exists: `og:title`, then fetcher frontmatter `title`, then ``. +2. Strip a leading `GitHub - `, so github.com repos become `<owner>/<repo>: <description>`, which matches existing posts. +3. Strip a trailing site/brand suffix after ` | `, ` – `, ` — ` or ` - ` when it names the site or publication rather than the article (e.g. ` | Crunchy Data Blog`, ` – site.com`). +4. Otherwise keep the wording, capitalization and punctuation verbatim; never translate. + +Summaries stay Vietnamese and are prose paragraphs only — no key-points bullet list. To insert an article safely, anchor the Edit on `### Bonus` and prepend: ``` @@ -92,10 +101,10 @@ node scripts/newsletter post-stats content/post/YYYY/MM/DD/index.md ``` Output (JSON): `{ post, newsletter, articles, images, videos, documents, total }`. -Render it as a single TL;DR line in the handler's report, omitting zero counts: +Render it as a single TL;DR line in the handler's report, omitting zero counts, with `N` from the `newsletter` field. This is the canonical format: the handlers and `mt-add-url` use it and do not restate it. ``` -📊 4 articles · 2 videos · 1 image +📊 Newsletter #N: 4 articles · 2 videos · 1 image ``` Use the counts the command returns — do not tally by hand. This is a **report-only** line: never write it into `index.md`. @@ -122,13 +131,22 @@ These rules apply only to text written into `content/post/**/index.md`. Keep use - Do not translate or localize article headings or image labels unless the user explicitly asks. - Video titles follow the `mt-add-video` handler's localization rules. +**Summary length:** article summaries are 1–2 prose paragraphs of at most 300 words, aiming for 280–298. A word is a whitespace-separated token, so each Vietnamese syllable counts as one. Check after inserting with an exact heading match: +```bash +awk -v h='## [Title](url)' '$0==h{f=1;next} /^##/{f=0} f{n+=NF} END{print n}' content/post/YYYY/MM/DD/index.md +``` +or before inserting with `printf '%s\n' "$SUMMARY" | awk '{n+=NF} END{print n}'`. Over 300 → trim and recount. Video blockquote summaries are 1–2 sentences. + **Content requirements:** audience = junior developers; tone = professional, clear, accessible; verify content is ≥99% Vietnamese before saving. ## 6. Error handling +This table owns the skip rules; handlers defer to it. + | Problem | Action | |-------|--------| -| URL inaccessible | Skip, note in report | -| Duplicate URL | Skip, note in report | -| Extraction fails | Skip, treat as error | +| Duplicate URL | Skip and name where it already is: `grep -rlF '<scheme://host/path>' content/post` (for Substack images, grep the image UUID instead), read that post's `title`, and report `duplicate (Newsletter #N, content/post/YYYY/MM/DD/index.md)` | +| Dead URL (404/410) | Skip immediately, note in report | +| `accessible: false` (403, 429, challenge, HEAD rejected, 000) | Not a skip — continue; for articles try WebFetch, then the `mt-fetch-url` chain | +| Extraction fails | Skip and report only after WebFetch and every `mt-fetch-url` tier fail | | Directory missing | Create directories | diff --git a/opencode.json b/opencode.json index 7eff406..fda40c5 100644 --- a/opencode.json +++ b/opencode.json @@ -5,7 +5,8 @@ "webfetch": "allow", "websearch": "allow", "bash": { - "node *": "allow", + "*": "ask", + "node scripts/newsletter *": "allow", "git status*": "allow", "git add*": "allow", "git commit*": "allow", @@ -19,9 +20,15 @@ "mv *": "allow", "mkdir *": "allow", "find *": "allow", + "find * -delete*": "ask", + "find * -exec*": "ask", "grep *": "allow", + "head *": "allow", + "sort*": "allow", + "awk *": "allow", + "wc *": "allow", "curl *": "allow", - "*": "ask" + "npx -y agent-browser *": "allow" } } }