fix(scripts): dedup URLs stored in markdown autolink form

The duplicate-check boundary set had '<' but not '>', so a URL stored
as <https://…> (autolink, used by older posts) was never detected as a
duplicate. Inherited from the JS engine; found while validating the Go
port against 2020-era posts.
This commit is contained in:
tiennm99 committed 2026-08-18 21:43:57 +07:00
1 parent e8451d8c24
commit c35a14f9bb
2 files changed
+18 -2

No files matched your search

@@ -34,3 +34,18 @@ Method: identical args to `node scripts/newsletter/<script>.js` and `go run ./sc
## Verdict
All matrix cases pass. Go engine is behavior-equivalent for skill consumption. Proceed to cutover.
## Post-cutover validation addendum (old-post URLs, 2026-08-18)
Ran the Go engine against URLs stored in old posts (2020 and 2025/02):
| Case | Result |
|------|--------|
| 2025/02 article, plain + tracker-variant | duplicate:true both, trackers stripped |
| 2025/02 YouTube stored watch URL + youtu.be variant | both canonicalize, duplicate:true, oEmbed title |
| 2025/02 S3 image (add-url + detect-image-source) | route:image, duplicate:true, uuid extracted |
| 18-month-old uuid via find-substack-post | `{found:false}` — correct, outside RSS window |
| 404 negative control | duplicate:false, http_status 404 |
| 2020 URL stored as markdown autolink `<https://…>` | **duplicate:false — defect found** |
Defect: the URL boundary set (`)]"'?#<_&,`, inherited verbatim from the JS) lacked `>`, so URLs stored in autolink form never dedup. Pre-existing JS behavior, not a port regression. Fixed post-cutover by adding `>` to the set in `url_utils.go`; regression-checked prefix-probe (still false), stored full URL (still true), deterministic outputs (unchanged).
+3 -2
View File
@@ -196,7 +196,8 @@ func uuidBoundaryOK(text string, end int) bool {
// urlBoundaryOK: an optional trailing slash (bareURL strips it, stored URLs may
// keep it), then a path/punctuation delimiter, whitespace, or end of text — so
// /p/foo does not match a stored /p/foo-bar.
// /p/foo does not match a stored /p/foo-bar. The '>' delimiter covers URLs
// stored in markdown autolink form <https://…>, which older posts use.
func urlBoundaryOK(text string, end int) bool {
if end < len(text) && text[end] == '/' {
end++
@@ -208,7 +209,7 @@ func urlBoundaryOK(text string, end int) bool {
case ' ', '\t', '\n', '\r', '\f', '\v':
return true
default:
return strings.IndexByte(`)]"'?#<_&,`, c) != -1
return strings.IndexByte(`)]"'?#<>_&,`, c) != -1
}
}