From f40103fe827a3d97570a625c62acc311b602effe Mon Sep 17 00:00:00 2001 From: tiennm99 Date: Sat, 30 May 2026 16:55:47 +0700 Subject: [PATCH] feat(skills): add mt-add-image handler with Substack source lookup Add the image handler in the meta-dispatch architecture. Given an image URL it resolves a label and inserts it into the newsletter Bonus -> Images. For Substack-hosted images, decode the S3 image UUID and find the source post across configured publications (ByteByteGo by default): RSS feed first, then a sitemap crawl up to ~3 months back. Label priority is figure caption, then post title, then a user-typed name. Non-Substack images are best-effort then ask. Publications are configurable and the skill offers to add a new one when the user identifies it. Wire mt-add-url to dispatch the image route to mt-add-image (previously a fallback). Update the project skills list. --- .claude/skills/mt-add-image/SKILL.md | 70 +++++++ .../config/substack-publications.json | 1 + .../scripts/detect-image-source.js | 60 ++++++ .../scripts/find-substack-post.js | 192 ++++++++++++++++++ .claude/skills/mt-add-url/SKILL.md | 10 +- CLAUDE.md | 3 +- 6 files changed, 331 insertions(+), 5 deletions(-) create mode 100644 .claude/skills/mt-add-image/SKILL.md create mode 100644 .claude/skills/mt-add-image/config/substack-publications.json create mode 100644 .claude/skills/mt-add-image/scripts/detect-image-source.js create mode 100644 .claude/skills/mt-add-image/scripts/find-substack-post.js diff --git a/.claude/skills/mt-add-image/SKILL.md b/.claude/skills/mt-add-image/SKILL.md new file mode 100644 index 0000000..f4feef2 --- /dev/null +++ b/.claude/skills/mt-add-image/SKILL.md @@ -0,0 +1,70 @@ +--- +name: mt-add-image +description: 'Image handler for the Hugo blog newsletter. Adds an image URL to today''s newsletter Bonus → Images with a detected label. For Substack-hosted images it finds the source post (ByteByteGo and other configured publications) via RSS then sitemap, using the figure caption or post title as the label; for other sites it is best-effort; if no label is found it asks you to type one. Normally invoked by the mt-add-url meta skill after classification, but can be used directly for a known image URL.' +--- + +## Overview + +`mt-add-image` is the **image handler**: given an image URL, it resolves a human-readable **label** and inserts `![label](image_url)` into today's newsletter **Bonus → Images**. It does not classify/route — that is `mt-add-url`'s job. + +Shared scripts: `.claude/skills/mt-add-url/scripts/`. Shared procedure: `../mt-add-url/references/newsletter-post-mechanics.md` — **follow it** for post find/create, Bonus insertion, and language rules. + +**Label priority:** figure caption → source post title → your typed input. + +## Input + +A clean image URL (passed by `mt-add-url`, or given directly). + +## Workflow + +### 1. Detect source +```bash +node .claude/skills/mt-add-image/scripts/detect-image-source.js "" +``` +→ `{ isSubstack, uuid?, innerUrl? }`. Skip the URL if it's a duplicate/inaccessible (the meta skill already checks; if invoked directly, sanity-check first). + +### 2a. Substack image (`isSubstack: true` with `uuid`) +Find the source post: +```bash +node .claude/skills/mt-add-image/scripts/find-substack-post.js --uuid +``` +- `found: true` → label = `caption` if non-empty, else `postTitle`. +- `found: false` → retry with the deeper sitemap crawl (slower — scans ~3 months, warn the user it may take ~15s): + ```bash + node .claude/skills/mt-add-image/scripts/find-substack-post.js --uuid --deep + ``` + - `found: true` → label as above. + - `found: false` → no automatic label; go to step 3 (ask). The result reports `scanned` + `cutoff` — mention how far back it looked. + +### 2b. Non-Substack image (`isSubstack: false`) +Best-effort only (no reverse-image-search): if you happen to know the page that contains the image, WebFetch it and read the OpenGraph title / nearby caption. Usually there's no containing page from just an image URL → go to step 3. + +### 3. Ask for a label (fallback) +If no label was detected, use `AskUserQuestion`: +- "Couldn't detect a name for this image (``). What label should I use?" (free-text). + +### 4. Add-publication offer (Substack miss only) +If a Substack image wasn't found and the user tells you which publication it's from, offer to append that host to `.claude/skills/mt-add-image/config/substack-publications.json`, then retry step 2a. This grows coverage for next time. + +### 5. Insert into Bonus → Images +Follow `../mt-add-url/references/newsletter-post-mechanics.md` to find/create today's post, then add under **Images**: +```markdown +**Images:** +![label](clean_image_url) +``` +Create `### Bonus` / `**Images:**` if missing. + +### 6. Report +``` +✅ Image added to Newsletter #[number] (Bonus → Images) +📄 content/post/YYYY/MM/DD/index.md +🏷️ label: "[label]" (source: rss | sitemap | opengraph | user-input) +🖼️ [clean_image_url] +``` + +## Checklist + +- [ ] Source detected (substack + uuid, or non-substack) +- [ ] Label resolved by priority (caption → post title → your input) +- [ ] Entry under Bonus → **Images** (section/subsection created if missing) +- [ ] Label localized to Vietnamese where natural (keep proper nouns) diff --git a/.claude/skills/mt-add-image/config/substack-publications.json b/.claude/skills/mt-add-image/config/substack-publications.json new file mode 100644 index 0000000..bfbfb04 --- /dev/null +++ b/.claude/skills/mt-add-image/config/substack-publications.json @@ -0,0 +1 @@ +["blog.bytebytego.com"] diff --git a/.claude/skills/mt-add-image/scripts/detect-image-source.js b/.claude/skills/mt-add-image/scripts/detect-image-source.js new file mode 100644 index 0000000..0b4878e --- /dev/null +++ b/.claude/skills/mt-add-image/scripts/detect-image-source.js @@ -0,0 +1,60 @@ +#!/usr/bin/env node +// Detect whether an image URL is Substack-hosted and extract its S3 image UUID. +// Usage: node detect-image-source.js "" +// Output: JSON { original_url, clean_url, isSubstack, uuid?, innerUrl? } +// +// Substack images are usually served via a CDN wrapper: +// https://substackcdn.com/image/fetch/$s_!x!,.../https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F_WxH.png +// The publication is NOT encoded in the URL — only the image identity (uuid) is. + +const path = require("path"); +const { cleanUrl } = require("../../mt-add-url/scripts/url-utils.js"); + +const url = process.argv[2]; +if (!url) { + console.error("Usage: node detect-image-source.js "); + process.exit(1); +} + +const SUBSTACK_HOSTS = ["substackcdn.com", "substack-post-media.s3.amazonaws.com"]; + +// Pull the inner S3 URL out of a substackcdn /image/fetch/ wrapper (if present). +function extractInnerUrl(targetUrl) { + const marker = targetUrl.indexOf("/https%3A%2F%2F"); + if (marker !== -1) { + return decodeURIComponent(targetUrl.slice(marker + 1)); + } + // Some forms embed a plain (already-decoded) inner https URL. + const plain = targetUrl.indexOf("/https://", 8); + if (plain !== -1) return targetUrl.slice(plain + 1); + return targetUrl; +} + +function detect(targetUrl) { + let host = ""; + try { + host = new URL(targetUrl).host.toLowerCase(); + } catch { + /* fall through — non-URL input is non-Substack */ + } + const innerUrl = extractInnerUrl(targetUrl); + const isSubstack = + SUBSTACK_HOSTS.includes(host) || /substack-post-media/i.test(innerUrl); + + // public/images/_WxH.ext — uuid is the stable image identity. + const m = innerUrl.match(/public\/images\/([0-9a-fA-F-]{36})/); + const uuid = m ? m[1].toLowerCase() : undefined; + + return { isSubstack, uuid, innerUrl: isSubstack ? innerUrl : undefined }; +} + +const cleaned = cleanUrl(url); +const { isSubstack, uuid, innerUrl } = detect(url); + +console.log(JSON.stringify({ + original_url: url, + clean_url: cleaned, + isSubstack, + ...(uuid ? { uuid } : {}), + ...(innerUrl ? { innerUrl } : {}), +}, null, 2)); diff --git a/.claude/skills/mt-add-image/scripts/find-substack-post.js b/.claude/skills/mt-add-image/scripts/find-substack-post.js new file mode 100644 index 0000000..88f0d78 --- /dev/null +++ b/.claude/skills/mt-add-image/scripts/find-substack-post.js @@ -0,0 +1,192 @@ +#!/usr/bin/env node +// Find which Substack post embeds a given image UUID, and extract a label. +// Usage: node find-substack-post.js --uuid [--deep] +// Output: JSON { found, source?, publication?, postTitle?, postUrl?, caption?, scanned? } +// +// Strategy: RSS feed first (fast, ~recent weeks). With --deep, fall back to a +// sitemap crawl up to ~3 months back (added in the deep-lookup phase). +// A Substack CDN URL does not encode its publication, so we search each +// publication listed in config/substack-publications.json. + +const fs = require("fs"); +const path = require("path"); + +const args = process.argv.slice(2); +function flag(name) { + const i = args.indexOf(name); + return i !== -1 ? (args[i + 1] && !args[i + 1].startsWith("--") ? args[i + 1] : true) : undefined; +} +const uuid = flag("--uuid"); +const deep = !!flag("--deep"); + +if (!uuid || uuid === true) { + console.error("Usage: node find-substack-post.js --uuid [--deep]"); + process.exit(1); +} + +const CONFIG = path.resolve(__dirname, "../config/substack-publications.json"); +function loadPublications() { + try { + return JSON.parse(fs.readFileSync(CONFIG, "utf-8")); + } catch { + return ["blog.bytebytego.com"]; + } +} + +async function fetchText(url) { + const controller = new AbortController(); + const timeout = setTimeout(() => controller.abort(), 10000); + try { + const res = await fetch(url, { + redirect: "follow", + signal: controller.signal, + headers: { "User-Agent": "Mozilla/5.0" }, + }); + return res.ok ? await res.text() : ""; + } catch { + return ""; + } finally { + clearTimeout(timeout); + } +} + +function decodeEntities(s) { + return s + .replace(/&/g, "&") + .replace(/</g, "<") + .replace(/>/g, ">") + .replace(/"/g, '"') + .replace(/�?39;|'/g, "'") + .replace(/ /g, " ") + .replace(/&#(\d+);/g, (_, n) => String.fromCharCode(+n)) + .trim(); +} + +function stripTags(html) { + return decodeEntities(html.replace(/<[^>]+>/g, "")).replace(/\s+/g, " ").trim(); +} + +// Pull the first (CDATA or plain) and <link> from an <item> chunk. +function itemTitle(item) { + const m = item.match(/<title>(?:<!\[CDATA\[)?([\s\S]*?)(?:\]\]>)?<\/title>/); + return m ? decodeEntities(m[1].trim()) : ""; +} +function itemLink(item) { + const m = item.match(/<link>(?:<!\[CDATA\[)?([\s\S]*?)(?:\]\]>)?<\/link>/); + return m ? m[1].trim() : ""; +} + +// If the UUID sits inside a <figure>…</figure>, return that figure's +// <figcaption> text. Cover images live in <enclosure> (no figure) → "". +function captionForUuid(item, id) { + const at = item.indexOf(id); + if (at === -1) return ""; + const figStart = item.lastIndexOf("<figure", at); + if (figStart === -1) return ""; + const figEnd = item.indexOf("</figure>", at); + if (figEnd === -1) return ""; + const figure = item.slice(figStart, figEnd); + const cap = figure.match(/<figcaption[^>]*>([\s\S]*?)<\/figcaption>/); + return cap ? stripTags(cap[1]) : ""; +} + +// Extract a post title from server-rendered post HTML (og:title preferred). +function postTitleFromHtml(html) { + const og = html.match(/<meta[^>]+property=["']og:title["'][^>]+content=["']([^"']+)["']/i); + if (og) return decodeEntities(og[1]); + const h1 = html.match(/<h1[^>]*>([\s\S]*?)<\/h1>/i); + if (h1) return stripTags(h1[1]); + const t = html.match(/<title>([\s\S]*?)<\/title>/i); + return t ? decodeEntities(t[1].trim()) : ""; +} + +// Deep fallback: crawl the sitemap back ~3 months, fetch posts most-recent-first +// (capped), and look for the UUID. Heavier than RSS — only used on RSS miss. +async function searchSitemap(publication, id, maxFetch = 40, monthsBack = 3) { + const xml = await fetchText(`https://${publication}/sitemap.xml`); + if (!xml) return { hit: null, scanned: 0, cutoff: null }; + + const cutoff = new Date(); + cutoff.setMonth(cutoff.getMonth() - monthsBack); + + const candidates = []; + for (const block of xml.split("<url>").slice(1)) { + const loc = block.match(/<loc>([^<]+)<\/loc>/); + const lastmod = block.match(/<lastmod>([^<]+)<\/lastmod>/); + if (!loc || !lastmod) continue; + if (!/\/p\//.test(loc[1])) continue; // posts only + const when = new Date(lastmod[1]); + if (isNaN(when) || when < cutoff) continue; + candidates.push({ url: loc[1], when }); + } + candidates.sort((a, b) => b.when - a.when); + + let scanned = 0; + for (const c of candidates.slice(0, maxFetch)) { + scanned++; + const html = await fetchText(c.url); + if (!html || !html.includes(id)) continue; + return { + hit: { + found: true, + source: "sitemap", + publication, + postTitle: postTitleFromHtml(html), + postUrl: c.url, + caption: captionForUuid(html, id), + }, + scanned, + cutoff: cutoff.toISOString().slice(0, 10), + }; + } + return { hit: null, scanned, cutoff: cutoff.toISOString().slice(0, 10) }; +} + +async function searchRss(publication, id) { + const xml = await fetchText(`https://${publication}/feed`); + if (!xml) return null; + const items = xml.split("<item>").slice(1); + for (const item of items) { + if (!item.includes(id)) continue; + const postTitle = itemTitle(item); + const caption = captionForUuid(item, id); + return { + found: true, + source: "rss", + publication, + postTitle, + postUrl: itemLink(item), + caption, + }; + } + return null; +} + +async function main() { + const publications = loadPublications(); + for (const pub of publications) { + const hit = await searchRss(pub, uuid); + if (hit) return console.log(JSON.stringify(hit, null, 2)); + } + + // Deep fallback: sitemap crawl up to ~3 months back. + if (deep) { + let totalScanned = 0; + let lastCutoff = null; + for (const pub of publications) { + const { hit, scanned, cutoff } = await searchSitemap(pub, uuid); + totalScanned += scanned; + lastCutoff = cutoff; + if (hit) return console.log(JSON.stringify(hit, null, 2)); + } + return console.log(JSON.stringify({ + found: false, + source: "sitemap", + scanned: totalScanned, + cutoff: lastCutoff, + }, null, 2)); + } + console.log(JSON.stringify({ found: false }, null, 2)); +} + +main(); diff --git a/.claude/skills/mt-add-url/SKILL.md b/.claude/skills/mt-add-url/SKILL.md index 5e86241..5b09cd4 100644 --- a/.claude/skills/mt-add-url/SKILL.md +++ b/.claude/skills/mt-add-url/SKILL.md @@ -10,8 +10,9 @@ description: 'Meta entry for adding URLs to the Hugo blog newsletter. Use whenev **Supported routes (this version):** - `article` → `mt-add-post` - `youtube` → `mt-add-video` +- `image` → `mt-add-image` -Everything else (`image`, direct `video` file, `document`, or anything unrecognized) is **not supported yet** → ask the user how to handle it. Image support is planned as a future `mt-add-image` skill. +Everything else (direct `video` file, `document`, or anything unrecognized) is **not supported yet** → ask the user how to handle it. ## Workflow @@ -37,7 +38,8 @@ Output (JSON): `{ original_url, clean_url, http_status, accessible, duplicate, r |-------|--------| | `article` | Invoke the **`mt-add-post`** skill, passing `clean_url` | | `youtube` | Invoke the **`mt-add-video`** skill, passing `clean_url` | -| `image` / `video` / `document` / anything else | **Fallback** — see step 4 | +| `image` | Invoke the **`mt-add-image`** skill, passing `clean_url` | +| `video` (direct file) / `document` / anything else | **Fallback** — see step 4 | Dispatch by calling the Skill tool for the chosen handler with `clean_url` as the argument. The handler completes the write end-to-end. @@ -45,11 +47,11 @@ Dispatch by calling the Skill tool for the chosen handler with `clean_url` as th ### 4. Fallback for unsupported types -When `route` is not `article` or `youtube`, do NOT write anything automatically. Use `AskUserQuestion`: +When `route` is not `article`, `youtube`, or `image`, do NOT write anything automatically. Use `AskUserQuestion`: - **Question:** "URL type `<route>` isn't supported yet (`<clean_url>`). How do you want to handle it?" - **Options:** - - "Add a new skill" — e.g. create `mt-add-image` for images. (Recommended for a type you'll reuse.) + - "Add a new skill" — e.g. a handler for this type. (Recommended for a type you'll reuse.) - "Update an existing skill" — extend a handler to support this URL type. - "Skip this URL" — leave it out of the newsletter. diff --git a/CLAUDE.md b/CLAUDE.md index 49f4dd4..d7d3e29 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -41,9 +41,10 @@ This project uses custom skills for automated workflows: - **mt-add-url**: Meta entry for adding URLs — classifies each URL and auto-dispatches to the right handler (the default entry when adding URLs to the newsletter) - **mt-add-post**: Article handler — adds an article/blog URL to the newsletter main content - **mt-add-video**: YouTube handler — adds a YouTube link to the newsletter Bonus → Videos +- **mt-add-image**: Image handler — adds an image to Bonus → Images; for Substack images finds the source post (ByteByteGo + configured publications) via RSS/sitemap to label it, else asks for a label - **mt-add-tags**: Add/update tags in Hugo post frontmatter -Shared scripts (`add-url.js`, `url-utils.js`, `find-newsletter-number.js`) and the shared post-mechanics reference live under `.claude/skills/mt-add-url/`. Image support is planned as a future `mt-add-image` handler; until then `mt-add-url` asks how to handle non-article/non-YouTube URLs. +Shared scripts (`add-url.js`, `url-utils.js`, `find-newsletter-number.js`) and the shared post-mechanics reference live under `.claude/skills/mt-add-url/`. `mt-add-url` dispatches `article`/`youtube`/`image`; other types (direct video files, documents, unknown) prompt the user to add or extend a handler. Refer to `.claude/skills/` for skill implementations.