The walk collects every item's file reference up front, and Telegram expires
those. The lifetime is undocumented; a run was seen to outlast an hour and fail
before two, which is less than a large chat spends downloading. So every fetch
past that point returned FILE_REFERENCE_EXPIRED, five in a row tripped the
breaker, and the run abandoned the rest of its todo list.
Telegram's contract is to cache a reference together with the source it came
from and re-read that source when it expires, so each message is re-read for a
live reference immediately before its own download. That is one round trip per
transfer and needs no guess at how long a reference lives. It runs after the
staging acquire rather than before: that acquire is the run's brake and blocks
for as long as the destination is slow, which on a stalled remote is long enough
to expire a fresh reference all over again.
A re-read that describes a different file — different name or different length —
is refused, because archiving it under the name this run recorded would store
the wrong bytes. That case, a deleted message, and one that no longer holds a
file are all skipped: nothing can be archived for them, and one must not strand
the rest of the chat.
Any other re-read failure is recorded as a failed download instead, because that
is what it is. The connection that serves the re-read is the one that serves the
file, so these arrive exactly when transfers are failing too. An item that never
reaches a worker never reaches OnAdd or OnDone, so on the skip path it would
move no counter, feed no streak and produce no outcome to retry — a source
outage would walk the whole list one dead round trip at a time and still exit as
merely incomplete.
tutil.GetSingleMessage is not reused for the re-read: it reports a deleted
message three different ways, two of them wrapping a nil error, and only one is
distinguishable — so deletions would be misclassified as transient and spend the
breaker's streak.
core's downloader logs a failed transfer and returns nil, and nothing here
installed a logger, so logctx handed out a nop and the reason was destroyed.
The size check in finish was all that survived, which reported a failed
two-gigabyte fetch as "short download: got 0 bytes" — a symptom with no cause
an operator can act on. The logger it reaches for now feeds a core that keeps
error entries and stores them on the element they belong to, so finish reports
the reason ahead of the byte count.
A failed download is also retried inside the run, up to three passes over
whatever is still missing and spaced minutes apart, because the failure this
repairs is transient: a dead connection takes every transfer in flight with it
and all of them are fetchable again afterwards, while leaving them to the next
run costs a full re-walk of the chat and a re-index of the remote first. An
item is counted once however many attempts it took, and a pass that recovers
reports nothing.
The download leg gains the breaker the upload leg already had. A source that
refuses everything refuses the rest of the list in milliseconds, so the pass
gives up after five failures in a row and the run exits 3 rather than spending
the entire outstanding list finding that out.
Real values from a live run had reached the README, the demo and several
fixtures: a chat name, its dialog id, Telegram file ids, filenames from the
chat, and the archive's own statistics. This repository archives a private
chat, so those name a real account and its contents, and history keeps them
long after the file is edited.
The values are gone from every commit. CLAUDE.md records the rule so they do
not come back, with a placeholder for each kind — a fixture keeps the property
it exists to test, so a multi-byte filename test still gets a multi-byte name,
just an invented one.
One combined figure could say how much had moved but not which half was
moving it. That is the question a slowing run actually raises: is Telegram
the bottleneck, or the remote? The two legs now have their own totals, files
and bytes each.
The gap between them is the useful part. They normally track a file or two
apart; a widening gap is the remote falling behind, which is also staging
filling up — visible now before the byte cap starts throttling downloads.
Uploads advance a whole file at a time because rclone's MoveFile is one
blocking call with no byte callbacks, so a partial upload counts for nothing
until it lands. That is the honest reading anyway: what the upload total
reports is what is actually on the remote.
Both renderers read the same counters, so the terminal and a captured log
cannot disagree about the numbers.
The shell pipeline's options were still accepted, so `sync -h` listed five
entries reading only "retired". The scripts themselves are gone, so the
migration hints have outlived what they pointed away from.
Each command now states what it does and which options are required, instead
of the flag package's bare "Usage of sync:". The top-level help lists the exit
codes, including the new 4, and says which one is worth retrying — that is the
part a driver script needs and the only place it was written down outside the
README.
Also fixes a real defect in the flag help: a backquoted word in a usage string
is the flag package's placeholder syntax, so "as `tdl dl --takeout` did"
rendered the option as "-takeout tdl dl --takeout". Backquotes are now used
deliberately, giving -c CHAT, -r REMOTE:PATH, -m SIZE and friends.
A run reported one aggregate line, which could say how much was done but never
what was happening: which files were moving, whether a stall was a slow
download or a slow upload, or how long the rest would take. On a transfer
measured in hours those are the only questions worth answering.
The pipeline now emits a per-item lifecycle rather than only Stats, and a
terminal renders it as an overall bar plus one bar per file in flight. Uploads
get a spinner rather than a bar because rclone's MoveFile is a single blocking
call with no byte callbacks; naming the file is still the point, since a run
that looks stalled is usually waiting on one large object.
Before the transfer, the run states what it found and what it will do. The
survey breaks the outstanding set down by reason — never fetched, zero-byte,
wrong size, unarchivable — because that is the difference between a run that
will converge and one that cannot, and the old "N to fetch" hid it. The plan
states the cap, staging path and concurrency, so a wrong setting is visible
before hours of transfer rather than after.
Redirected output keeps plain lines and gains one per archived file. Bars are
continuous cursor movement, and a captured log of them is what tdl's progress
bar did to the shell pipeline's logs.
cmd/uidemo renders the whole thing against fake data. It is how the layout was
checked without a session, and it earned its place immediately: the overall bar
sat at zero because Stats only tracked the file count and never advanced it.
core's Download returns without waiting on its worker group when the
iterator reports an error, so surfacing one through Iter.Err left workers
sending into a channel the caller had already closed. The iterator now
always reports a nil error and stashes the real one, read after Download
returns.
The circuit breaker cancelled only the upload context, which left
downloads running full speed against a remote refusing them: every file
stayed in staging and every reservation came back, so a broken remote
filled local disk faster than a working one. It now stops the download
iterator instead.
A failed move leaves the local copy in place, so the byte reservation
cannot be handed back until the file is removed. A confirmed-short object
is deleted rather than left under a name verification would count as
archived forever.
Also: release the reservation when opening the staging file fails, report
results even when an upload errored, bound the recorded errors, and drop
the reporter's lock before writing so terminal latency cannot throttle
downloads.
Backends that finish an upload as a server-side task, pikpak among them,
leave it pending long enough that rclone abandons the transfer once
--low-level-retries polls run out. Default RCLONE_TRANSFERS to 2 to keep
that task queue short and RCLONE_LOW_LEVEL_RETRIES to 20 to wait a slow
commit out. Both use the :=default form, so an exported value still wins.
Shorten the --min-age default to 45s: excluding '*.tmp' is what actually
keeps an unfinished download from being uploaded, so the age guard only
needs to cover a rename racing a sweep, and two minutes of dwell made
staging larger than it had to be.
Sweep every 120s rather than 300s in the driver, which shrinks the backlog
each sweep has to move, and raise the pass limit to 30 to suit chats that
need more rounds than 20 to converge.
Add -m SIZE to run.sh, passed through by export-until-complete.sh, so the
staging directory cannot outgrow the local disk when tdl downloads faster
than rclone uploads. Staging is measured every 10 seconds, independently of
the sweep interval; at the cap tdl is suspended with SIGSTOP and rclone
sweeps until staging is back under it, then tdl resumes and --continue picks
its .tmp files back up. The drain loops while it frees space and gives up
once only in-flight .tmp files remain, which no sweep can move, so a cap set
below what the concurrent downloads hold warns instead of stalling. The
failure streak still counts only sweeps that actually ran, so a quiet cap
check cannot clear it.
The sweeps that run once at the end now report progress rather than going
quiet for minutes: rclone's redrawn bar on a terminal, one-line stats every
30s when output is redirected. Stats are logged at INFO, so only the stats
level is raised to NOTICE, avoiding the line-per-file output -v would add.
Also send SIGCONT before SIGTERM when stopping tdl, since a suspended
process never sees the term.
run.sh finishes when tdl finishes, which does not mean every file arrived: a
dropped session, a stalled remote or an interrupted pass leaves gaps that
nothing reported.
verify-export.sh rebuilds the filename tdl produces for each media message in
the export JSON and checks the remote and staging for it. Messages carrying no
media are skipped rather than counted as gaps. A zero-byte file counts as
missing because rclone overwrites a size-mismatched destination, so a retry
repairs it; files under 1 KiB are reported but not retried, since real media is
sometimes that small and retrying them would never terminate.
export-until-complete.sh loops that check with run.sh, narrowing the export to
the outstanding ids each pass. It stops when the verifier reports complete, when
a pass fetches nothing new, when the remote runs low on space, or on Ctrl-C.
tdl's progress bar is shown on a terminal and suppressed when output is
redirected, so logs stay readable.
Exports, staging and logs are now ignored: they hold real message ids, file
names and text from an account.
-c already passed through to tdl, which resolves a numeric argument as an
MTProto id and anything else through gotd's resolver (@name, name, t.me and
tg:// links). Two forms needed handling: a Bot API '-100' prefixed id, which
MTProto does not use and which tdl never strips, is now converted; and a
message link, which names a message rather than a chat, is rejected with a
clear error instead of a resolver failure.
The export JSON now defaults to export-<chat>.json when -c is given, so
exporting a second chat from the same directory cannot silently download the
first chat again from a stale export.json. An existing file is still reused,
but the script now says which file it is using.
The pipeline never depended on WebDAV; only the naming, docs and preflight
did. Replace the WebDAV-root listing with a portable check: a named remote
must appear in 'rclone listremotes', and creating the destination proves
reachability and credentials. On-the-fly connection strings have no config
entry, so the name check is skipped for them.
Read the remote list into a variable rather than piping it into 'grep -q':
grep exits on the first match and kills rclone with SIGPIPE, which pipefail
reports as failure, intermittently rejecting a configured remote.
Document that rclone's flags are all settable through their RCLONE_*
environment variables, so the upload side is tunable without new options.
The README now covers only what the tool does and how to run it: setup,
options, tdl flag pass-through, subset exports, resume semantics, the
failure modes it guards against, and exit codes.
Flags are documented from tdl's own source rather than assumed: dl -f takes
the exported JSON while dl -i/-e are file-extension filters, and the
expression filter is chat export -f. Chats are addressed by id or domain.
Also warn that the inline rclone loops need --exclude '*.tmp' themselves.
Without it a download stalled by a flood wait is uploaded half-written and
its local copy deleted, which breaks tdl --continue for that file.
The Python/Telethon exporter is retired in favor of iyear/tdl, which
covers the same purpose and is actively maintained. README now carries
only migration instructions, including the rclone rolling-pipeline
recipe for exporting to WebDAV. Full implementation remains at d170986.
Two README statements were false rather than merely incomplete. "--dry-run
takes no lock at all, so it can always be run against an export that is
currently in progress" contradicted the paragraph four lines above it
warning that two clients sharing one session is a corruption hazard:
dry-run opens the session file like any other run. Both places now say
what is true - no *export* lock, but a session lock, so an alongside
dry-run needs its own --session.
The exit table promised 1 for a bad argument. argparse exits 2 on its
own, before main()'s try block can map anything, so a bad --limit
collided with the dry-run SHORT verdict. Documented as the shared code
it is instead of a code the tool never returns.
Also notes that both locks are flock-based, and that flock is advisory or
per-client on NFS, so single-instance enforcement is not guaranteed
there. Setup gains the venv creation step it assumed.
Phase 7 gains step 12b: measure the sweep cost under a date filter before
deciding whether to optimize it. With reverse=True and no offset_date,
telethon starts the sweep at message id 1 - verified in the pinned wheel -
so --since walks the whole history discarding messages in keep(), and
--until never terminates early. Neither fix is free: offset_date can
start the sweep mid-album, which would take a post_id that is not the
album's lowest id and break filter-invariant identity, and an --until
break assumes dates rise with ids, false for imported history. The
swept-to-kept ratio decides whether either risk is worth taking.
Reports from the review pass are recorded under plans/reports/, including
the findings left unfixed: resolve_entity catching only ValueError,
_find_in_dialogs being the one network loop outside a flood primitive,
SystemExit in session.py routing around the Abort contract, and
completed_at being cleared before any work begins.
Covers why the Bot API structurally cannot do this, what .env may and
may not hold, why the export directory is a chat id, and the sidecar's
union-over-records rule - reading it as last-write-wins reports a
narrower file list than what is on disk after a filter change.
Also states plainly that flood waits have no ceiling by default: the
tool sleeps as long as Telegram demands and logs the computed wake time,
so a long sleep reads as a sleep rather than a hang.
tiennm99