mirror of
https://github.com/tiennm99/blog.git
synced 2026-10-14 00:16:34 +00:00
fix(scripts): dedup URLs stored in markdown autolink form
The duplicate-check boundary set had '<' but not '>', so a URL stored as <https://…> (autolink, used by older posts) was never detected as a duplicate. Inherited from the JS engine; found while validating the Go port against 2020-era posts.
This commit is contained in:
1 parent
10272495f5
commit
1794b68538
2 files changed
+18
-2
No files matched your search
@@ -34,3 +34,18 @@ Method: identical args to `node scripts/newsletter/<script>.js` and `go run ./sc
|
||||
## Verdict
|
||||
|
||||
All matrix cases pass. Go engine is behavior-equivalent for skill consumption. Proceed to cutover.
|
||||
|
||||
## Post-cutover validation addendum (old-post URLs, 2026-08-18)
|
||||
|
||||
Ran the Go engine against URLs stored in old posts (2020 and 2025/02):
|
||||
|
||||
| Case | Result |
|
||||
|------|--------|
|
||||
| 2025/02 article, plain + tracker-variant | duplicate:true both, trackers stripped |
|
||||
| 2025/02 YouTube stored watch URL + youtu.be variant | both canonicalize, duplicate:true, oEmbed title |
|
||||
| 2025/02 S3 image (add-url + detect-image-source) | route:image, duplicate:true, uuid extracted |
|
||||
| 18-month-old uuid via find-substack-post | `{found:false}` — correct, outside RSS window |
|
||||
| 404 negative control | duplicate:false, http_status 404 |
|
||||
| 2020 URL stored as markdown autolink `<https://…>` | **duplicate:false — defect found** |
|
||||
|
||||
Defect: the URL boundary set (`)]"'?#<_&,`, inherited verbatim from the JS) lacked `>`, so URLs stored in autolink form never dedup. Pre-existing JS behavior, not a port regression. Fixed post-cutover by adding `>` to the set in `url_utils.go`; regression-checked prefix-probe (still false), stored full URL (still true), deterministic outputs (unchanged).
|
||||
@@ -196,7 +196,8 @@ func uuidBoundaryOK(text string, end int) bool {
|
||||
|
||||
// urlBoundaryOK: an optional trailing slash (bareURL strips it, stored URLs may
|
||||
// keep it), then a path/punctuation delimiter, whitespace, or end of text — so
|
||||
// /p/foo does not match a stored /p/foo-bar.
|
||||
// /p/foo does not match a stored /p/foo-bar. The '>' delimiter covers URLs
|
||||
// stored in markdown autolink form <https://…>, which older posts use.
|
||||
func urlBoundaryOK(text string, end int) bool {
|
||||
if end < len(text) && text[end] == '/' {
|
||||
end++
|
||||
@@ -208,7 +209,7 @@ func urlBoundaryOK(text string, end int) bool {
|
||||
case ' ', '\t', '\n', '\r', '\f', '\v':
|
||||
return true
|
||||
default:
|
||||
return strings.IndexByte(`)]"'?#<_&,`, c) != -1
|
||||
return strings.IndexByte(`)]"'?#<>_&,`, c) != -1
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user