The walk collects every item's file reference up front, and Telegram expires
those. The lifetime is undocumented; a run was seen to outlast an hour and fail
before two, which is less than a large chat spends downloading. So every fetch
past that point returned FILE_REFERENCE_EXPIRED, five in a row tripped the
breaker, and the run abandoned the rest of its todo list.
Telegram's contract is to cache a reference together with the source it came
from and re-read that source when it expires, so each message is re-read for a
live reference immediately before its own download. That is one round trip per
transfer and needs no guess at how long a reference lives. It runs after the
staging acquire rather than before: that acquire is the run's brake and blocks
for as long as the destination is slow, which on a stalled remote is long enough
to expire a fresh reference all over again.
A re-read that describes a different file — different name or different length —
is refused, because archiving it under the name this run recorded would store
the wrong bytes. That case, a deleted message, and one that no longer holds a
file are all skipped: nothing can be archived for them, and one must not strand
the rest of the chat.
Any other re-read failure is recorded as a failed download instead, because that
is what it is. The connection that serves the re-read is the one that serves the
file, so these arrive exactly when transfers are failing too. An item that never
reaches a worker never reaches OnAdd or OnDone, so on the skip path it would
move no counter, feed no streak and produce no outcome to retry — a source
outage would walk the whole list one dead round trip at a time and still exit as
merely incomplete.
tutil.GetSingleMessage is not reused for the re-read: it reports a deleted
message three different ways, two of them wrapping a nil error, and only one is
distinguishable — so deletions would be misclassified as transient and spend the
breaker's streak.
core's downloader logs a failed transfer and returns nil, and nothing here
installed a logger, so logctx handed out a nop and the reason was destroyed.
The size check in finish was all that survived, which reported a failed
two-gigabyte fetch as "short download: got 0 bytes" — a symptom with no cause
an operator can act on. The logger it reaches for now feeds a core that keeps
error entries and stores them on the element they belong to, so finish reports
the reason ahead of the byte count.
A failed download is also retried inside the run, up to three passes over
whatever is still missing and spaced minutes apart, because the failure this
repairs is transient: a dead connection takes every transfer in flight with it
and all of them are fetchable again afterwards, while leaving them to the next
run costs a full re-walk of the chat and a re-index of the remote first. An
item is counted once however many attempts it took, and a pass that recovers
reports nothing.
The download leg gains the breaker the upload leg already had. A source that
refuses everything refuses the rest of the list in milliseconds, so the pass
gives up after five failures in a row and the run exits 3 rather than spending
the entire outstanding list finding that out.
A run reported one aggregate line, which could say how much was done but never
what was happening: which files were moving, whether a stall was a slow
download or a slow upload, or how long the rest would take. On a transfer
measured in hours those are the only questions worth answering.
The pipeline now emits a per-item lifecycle rather than only Stats, and a
terminal renders it as an overall bar plus one bar per file in flight. Uploads
get a spinner rather than a bar because rclone's MoveFile is a single blocking
call with no byte callbacks; naming the file is still the point, since a run
that looks stalled is usually waiting on one large object.
Before the transfer, the run states what it found and what it will do. The
survey breaks the outstanding set down by reason — never fetched, zero-byte,
wrong size, unarchivable — because that is the difference between a run that
will converge and one that cannot, and the old "N to fetch" hid it. The plan
states the cap, staging path and concurrency, so a wrong setting is visible
before hours of transfer rather than after.
Redirected output keeps plain lines and gains one per archived file. Bars are
continuous cursor movement, and a captured log of them is what tdl's progress
bar did to the shell pipeline's logs.
cmd/uidemo renders the whole thing against fake data. It is how the layout was
checked without a session, and it earned its place immediately: the overall bar
sat at zero because Stats only tracked the file count and never advanced it.
Phases 4 through 6: the two legs and the command that joins them.
Downloads go to <name>.part and are renamed only once complete, so a file
without the suffix is always whole. That is what lets the upload leg treat
"exists" as "finished" — the property run.sh could only approximate with a
filename convention plus an age guard, because it could not see inside tdl.
Every finished file is checked against the size Telegram reported, and that
check rather than the error is the authoritative signal. core's downloader logs
a failed transfer and returns nil, and its completion callback is deferred on
that named return, so a failure arrives indistinguishable from a success.
Trusting it would promote a truncated file and archive it as complete.
The disk cap is a semaphore over bytes. A download reserves its own size before
starting and releases it only after the upload confirms, so a slow remote
stalls downloads by itself. Blocking the iterator is safe because the
downloader calls it from its dispatch loop while workers run in a group, so a
blocked iterator never stops the uploads that free the space. Gone with it: the
du polling, the SIGSTOP and SIGCONT suspension, the min-age guard, the
temp-file filter and the sweep-failure counter.
A cap smaller than the largest file is refused up front. The semaphore could
never admit it, and a run blocked on a file it can never start looks exactly
like a stalled remote.
Uploads re-state each object to prove its size before the local copy is gone,
closing a gap where a truncated upload was only noticed by a later verify.
The destination is created before the chat is read. It is also the credentials
check, and doing it first means a bad destination fails in seconds rather than
after a full history walk.
One invocation converges: each item is checked against the index immediately
before download, so there are no passes and re-running is the resume path.
Options that no longer exist say what replaced them instead of failing as
unknown flags.
Verified end to end against the live chat and a scratch remote path: two files
downloaded, uploaded, confirmed present at the right size, staging left empty.
tiennm99