rclone does not address a file by the bytes os.OpenFile wrote. Names handed to
an Fs go through the backend encoder and names listed back are re-encoded to
the standard set, neither of which the write path performs. A filename holding
one of the rewritten characters was therefore stored under one string and
looked up under another: confirmed against the local backend, where a written
"a‛b.jpg" reports object not found and a written "a\nb.jpg" lists back as
"a␊b.jpg". That is the same two-derivation divergence this program was written
to remove, with rclone's encoder standing where filenamify used to. Such names
are rejected, not encoded, for the same reason every other name is.
The length limit ignored the ".part" suffix that is opened first, so a name
just inside NAME_MAX passed the check and then failed to open on every pass,
stalling the walk on that message forever. The suffix now lives beside the
limit that has to account for it.
filter.NewFilter(nil) does not build a neutral filter; it copies the package
global, which rclone has already filled from RCLONE_*. Indexing inherited the
operator's environment, so a stray RCLONE_MIN_SIZE emptied the index and
re-downloaded the archive. Every narrowing field is now set explicitly and the
result is asserted inactive. A subprocess test covers it, since the env is read
at package init and t.Setenv is too late to observe anything.
An ErrorDirNotFound from a subdirectory was also treated as an empty
destination, returning a partial index as authoritative.
t.me/c/<id> and t.me/s/<name> were passed through whole, and gotd reads the
first path component as the username — resolving "c" or "s", which is a
confusing failure at best and someone else's chat at worst, since
one-character usernames exist. Both now yield the chat, and t.me/s/<name>/<id>
is refused like any other message link. Two tests asserted the old behaviour.
core's dcpool.Takeout deadlocks when takeout init fails: it holds the pool
mutex and recovers by calling Client, which takes the same non-reentrant
mutex. Telegram returns TAKEOUT_INIT_DELAY for a takeout started recently and
takeout is on by default, so two runs in succession hang the process with no
output and no response to cancellation. The session is established once here
instead, falling back to a plain client, and the pool's own Takeout is never
called.
Third slice: verify-export.sh, without the subprocess or the python.
One rclone listing builds an in-memory index, and the report is computed from
it. missing-ids.txt and gap.json are gone; so is every python3 heredoc.
Presence is answered from a whole name and never from a message id. The
id-keyed map exists only to tell "absent" apart from "absent, but a stale copy
under an older name is sitting there", and it stays unexported so nothing can
reach for it as an answer. That distinction is the bug this rewrite exists to
remove, so it is enforced by structure rather than by comment.
Names are checked for path containment before use. Storing them verbatim means
a filename chosen by whoever uploaded the file can contain a separator or a
parent reference, and tdl never had to care because its template rewrote those
away. Over-long names are refused for the same reason: the filesystem would
reject them at create time, and a file that can never be written would be
reported absent on every pass forever.
Objects are addressed by the path rclone knows them by, not by the basename
used for matching. The two differ once a remote has directory structure, and
deleting by basename would miss the object or remove a same-named one from the
root. Basenames appearing at more than one path make the snapshot ambiguous, so
they are reported rather than silently resolved.
Indexing runs at full depth with filters cleared. Inheriting RCLONE_MAX_DEPTH
or RCLONE_EXCLUDE would not fail, it would quietly report archived files as
absent and fetch them all again.
Deleting stale copies stays opt-in and confirmed; a non-interactive stdin
declines rather than proceeding. Filenames are quoted wherever they are
printed, so an embedded escape cannot redraw the list an operator approves.
Verified against the live remote: identical to verify-export.sh on the same
state — 18155 expected, 15548 present, 2607 absent, ids 9857-18013, exit 1.
Second slice: the read path, and the fix for the bug that motivated the
rewrite.
The shell pipeline derived a filename twice. `tdl chat export` wrote the raw
Telegram name into a JSON, while `tdl dl` rendered it through a template whose
default pipes it through filenamify, which rewrites reserved characters and
collapses runs of '!'. A message whose name contained '!!' was therefore looked
up under one name and stored under another; the verifier never found it and
re-fetched it on every pass. Here a single function produces the name, and the
string it returns is used both to test for presence and to write the file, so
the two cannot disagree.
Names are stored exactly as Telegram reports them rather than reproducing
filenamify. That is a deliberate break from what the old pipeline wrote: a file
it stored under a rewritten name is not recognised and will be fetched again.
For the one chat archived so far that is a single file out of 18155, already
removed. Because names are verbatim they are not path-safe, so the code that
turns one into a path must enforce containment.
Walk yields a sequence rather than taking a callback, since the downloader
consumes a pull iterator and range-over-func converts either way without anyone
owning a goroutine. It pages newest first, where the old pipeline went oldest
first, which changes what an interrupted run leaves behind.
Message links are refused rather than guessed at, across every host Telegram
uses and the tg:// forms that carry the message id in a query parameter. A
private channel link and a public message link have the same shape, so the two
are told apart by parsing rather than by pattern.
Verified against the live chat: 18155 media messages, matching the shell
verifier, and every one of the 15548 objects already on the remote is found
under a derived name.
tiennm99