tiennm99
baab3d443a
Switches every call site to `node scripts/newsletter` and, in the same pass, fixes three structural problems in the skill layer. The seven subcommands were spelled out in five files; they are now enumerated once in docs/newsletter/engine-commands.md, and every other mention is a link except where a skill inlines the one or two commands it actually invokes. The shared post mechanics move out of mt-add-url's directory into docs/newsletter/, so no skill owns another skill's documentation and both runtimes reach it by a repository-relative path. The Codex adapters become symlinks to their canonical counterparts, which removes the parallel frontmatter that could drift. Two skills are renamed for a uniform mt-<verb>-<object> scheme: mt-add-post becomes mt-add-article, since it adds an article to a post, and mt-webfetch becomes mt-fetch-url. The other four already conformed. Note that $mt-add-post and mt-webfetch no longer resolve; no alias directories are left behind, because two names for one skill is the duplication this change removes. Trigger wording in the frontmatter descriptions is left alone apart from the renamed tokens and the corrected fetch-tier summary: those strings are how the runtimes decide whether to invoke a skill. Setup documentation now states the npm ci prerequisite. The engine has dependencies, so unlike its predecessor it does not run on a bare clone.
4.9 KiB
Newsletter engine commands
The portable newsletter engine lives in scripts/newsletter/ — Node ESM, one
module per subcommand. This file is the only place the commands are
enumerated; skills and docs link here rather than restating the list.
Prerequisite
The engine has three runtime dependencies (cheerio, defuddle, linkedom), so
a fresh clone needs one setup step:
npm ci
Skip it and the content-parsing subcommands print
newsletter engine: dependencies missing — run 'npm ci' from the repo root
and exit 1 rather than a module-resolution stack trace. fetch-via-defuddle is
the exception: its local tier needs the dependencies, so without them it falls
through to the defuddle.md proxy and still succeeds.
Invocation
Always from the repo root — repo-relative paths (content/post) resolve from the
process working directory:
node scripts/newsletter <command> [args]
JSON output is 2-space indented with a trailing newline, and is not HTML-escaped
(URLs keep their & readable). Optional fields are omitted when empty rather
than emitted as "".
Commands
add-url <url>
Classifies and deduplicates one URL — the single entry point per URL.
{
"original_url": "…",
"clean_url": "…",
"http_status": "200",
"accessible": true,
"duplicate": false,
"route": "article",
"title": "…",
"author": "…"
}
route∈youtube | image | video | document | articleclean_urlhas tracking parameters stripped (utm_*,fbclid,gclid,ref, …); surviving parameters keep their original order and encoding- For
youtube,clean_urlis the canonicalhttps://www.youtube.com/watch?v=IDandtitle/authorcome from oEmbed. Both keys are absent for every other route, and absent when oEmbed fails. duplicateis boundary-aware: a URL that is merely a prefix of one already stored is not a duplicate. Substack images dedupe on their image uuid, so transform and size variants collapse onto one identity.
find-newsletter-number
Prints the next newsletter number (plain text). Scans content/post/YYYY/MM/DD/
newest-year-first and stops at the first year that holds a Newsletter #N
heading, so a stale higher number in an older year does not win. An empty tree
yields 1.
list-existing-tags
Prints tag frequencies, most-used first, top 40, as %6d %s — the count
right-aligned in six columns. Ties keep first-seen (lexical walk) order.
detect-image-source <url>
Detects a Substack-hosted image and extracts its S3 image uuid.
{
"original_url": "…",
"clean_url": "…",
"isSubstack": true,
"uuid": "…",
"innerUrl": "…"
}
uuid and innerUrl are present only when the URL is a Substack image and the
value is non-empty. A malformed percent sequence in the CDN wrapper falls back to
the raw substring rather than failing.
find-substack-post --uuid <uuid> [--deep]
Finds which Substack post embeds an image uuid, and extracts a label. Only the double-dash flag form is accepted.
Searches each publication in scripts/newsletter/config/substack-publications.json
(editable, read at runtime) — the RSS feed first, then, with --deep, a sitemap
crawl back ~3 months sharing a 40-fetch budget across all publications.
On a hit:
{
"found": true,
"source": "rss",
"publication": "…",
"postTitle": "…",
"postUrl": "…",
"caption": "…",
"candidates": ["…"]
}
On a miss: {"found": false}, or with --deep
{"found": false, "source": "sitemap", "scanned": 0, "budget": 40, "cutoff": null}
where cutoff is the crawl boundary date, or null when no sitemap could be
fetched at all.
fetch-via-defuddle <url>
Fallback fetcher for pages that blocked the built-in fetch. Two tiers, in order:
local defuddle extraction, then the defuddle.md proxy (which fetches from its
own IP — the tier that matters when this machine's IP is the blocked one).
Both tiers write YAML frontmatter followed by the markdown body, so the caller parses one shape either way; stderr names which tier failed and why. An extraction that looks like a bot challenge rather than the requested page counts as a local-tier failure, so the proxy still gets its turn.
| Exit | Meaning |
|---|---|
| 0 | content returned |
| 1 | every tier failed |
| 2 | bad arguments |
post-stats <path/to/index.md>
Counts what a post already holds, so a handler can report a running tally.
{
"post": "content/post/2026/09/10/index.md",
"newsletter": 132,
"articles": 8,
"images": 3,
"videos": 1,
"documents": 0,
"total": 12
}
newsletter is 0 when the post carries no Newsletter #N heading.
Tests
npm test # deterministic tier, offline, ~2s
NEWSLETTER_NET=1 npm test # adds the live-upstream tier
Golden outputs under scripts/newsletter/__fixtures__/golden/ were captured from
the previous Go implementation; see
plans/reports/parity-260918-newsletter-js-migration-report.md.