Both phases ran for minutes printing nothing between their opening line and
their result, so a working run looked exactly like a hung one — which is how it
was reported.
The message counter lives in Walk rather than in the caller's loop because most
of a chat is not media: text-only and service messages are filtered out inside
the walk, so a caller counting yielded items still sees nothing while crossing a
long stretch of conversation.
Cadence follows the existing reporter: a terminal redraws one line, a redirected
run gets a periodic one, since ANSI redraws are what turned the shell pipeline's
captured logs into megabytes of control characters.
Verification matched on name and non-zero size, so an upload that died partway
was counted archived permanently. Comparing against the size Telegram reports
found six such objects in the live archive, one of them 221 MiB standing in for
a 2006 MiB video. They are folded into the outstanding set; rclone overwrites a
size mismatch, so another pass repairs them.
A zero Report claimed the archive was complete — nothing expected, nothing
missing — which is the value both commands hold before their Telegram callback
populates it, so any early return printed COMPLETE and exited 0 on an untouched
chat. A Report now knows whether it ran.
A name that can never be written kept the run outstanding forever while the
download set deliberately excluded it, so a driver looping on "incomplete"
walked the whole history and re-indexed the whole remote every pass for work
that could not be done. Such a run now reports STALLED and exits 4.
A destination that stopped accepting uploads was reported and then discarded,
exiting 1. The same driver would retry against a full or unreachable remote
indefinitely, downloading gigabytes each pass to upload none. It exits 3.
Free space is re-checked during the run, not only before it. An archive this
size runs for hours, and the remote can fill in the middle; discovering it
through five failed multi-gigabyte uploads wastes the download for all of them.
Also: parseSize silently wrapped to a negative or zero on a large input, which
reads downstream as "no cap"; list printed attacker-chosen filenames raw, so a
tab shifted the columns and an escape sequence reached the terminal; a missing
backend blamed credentials rather than the build; humanBytes indexed past its
unit table above 1 PiB; a second Init reported success against a config that
never loaded; and sync did not surface the basename collisions verify warned
about, though sync is the command that acts on the verdict.
The env-override test could not observe what it claimed: rclone reads RCLONE_*
at package init, so t.Setenv came too late and the assertion held with the
guard removed. It runs in a subprocess now, as does the new one covering the
index against inherited filters.
rclone does not address a file by the bytes os.OpenFile wrote. Names handed to
an Fs go through the backend encoder and names listed back are re-encoded to
the standard set, neither of which the write path performs. A filename holding
one of the rewritten characters was therefore stored under one string and
looked up under another: confirmed against the local backend, where a written
"a‛b.jpg" reports object not found and a written "a\nb.jpg" lists back as
"a␊b.jpg". That is the same two-derivation divergence this program was written
to remove, with rclone's encoder standing where filenamify used to. Such names
are rejected, not encoded, for the same reason every other name is.
The length limit ignored the ".part" suffix that is opened first, so a name
just inside NAME_MAX passed the check and then failed to open on every pass,
stalling the walk on that message forever. The suffix now lives beside the
limit that has to account for it.
filter.NewFilter(nil) does not build a neutral filter; it copies the package
global, which rclone has already filled from RCLONE_*. Indexing inherited the
operator's environment, so a stray RCLONE_MIN_SIZE emptied the index and
re-downloaded the archive. Every narrowing field is now set explicitly and the
result is asserted inactive. A subprocess test covers it, since the env is read
at package init and t.Setenv is too late to observe anything.
An ErrorDirNotFound from a subdirectory was also treated as an empty
destination, returning a partial index as authoritative.
t.me/c/<id> and t.me/s/<name> were passed through whole, and gotd reads the
first path component as the username — resolving "c" or "s", which is a
confusing failure at best and someone else's chat at worst, since
one-character usernames exist. Both now yield the chat, and t.me/s/<name>/<id>
is refused like any other message link. Two tests asserted the old behaviour.
core's dcpool.Takeout deadlocks when takeout init fails: it holds the pool
mutex and recovers by calling Client, which takes the same non-reentrant
mutex. Telegram returns TAKEOUT_INIT_DELAY for a takeout started recently and
takeout is on by default, so two runs in succession hang the process with no
output and no response to cancellation. The session is established once here
instead, falling back to a plain client, and the pool's own Takeout is never
called.
Phases 4 through 6: the two legs and the command that joins them.
Downloads go to <name>.part and are renamed only once complete, so a file
without the suffix is always whole. That is what lets the upload leg treat
"exists" as "finished" — the property run.sh could only approximate with a
filename convention plus an age guard, because it could not see inside tdl.
Every finished file is checked against the size Telegram reported, and that
check rather than the error is the authoritative signal. core's downloader logs
a failed transfer and returns nil, and its completion callback is deferred on
that named return, so a failure arrives indistinguishable from a success.
Trusting it would promote a truncated file and archive it as complete.
The disk cap is a semaphore over bytes. A download reserves its own size before
starting and releases it only after the upload confirms, so a slow remote
stalls downloads by itself. Blocking the iterator is safe because the
downloader calls it from its dispatch loop while workers run in a group, so a
blocked iterator never stops the uploads that free the space. Gone with it: the
du polling, the SIGSTOP and SIGCONT suspension, the min-age guard, the
temp-file filter and the sweep-failure counter.
A cap smaller than the largest file is refused up front. The semaphore could
never admit it, and a run blocked on a file it can never start looks exactly
like a stalled remote.
Uploads re-state each object to prove its size before the local copy is gone,
closing a gap where a truncated upload was only noticed by a later verify.
The destination is created before the chat is read. It is also the credentials
check, and doing it first means a bad destination fails in seconds rather than
after a full history walk.
One invocation converges: each item is checked against the index immediately
before download, so there are no passes and re-running is the resume path.
Options that no longer exist say what replaced them instead of failing as
unknown flags.
Verified end to end against the live chat and a scratch remote path: two files
downloaded, uploaded, confirmed present at the right size, staging left empty.
Third slice: verify-export.sh, without the subprocess or the python.
One rclone listing builds an in-memory index, and the report is computed from
it. missing-ids.txt and gap.json are gone; so is every python3 heredoc.
Presence is answered from a whole name and never from a message id. The
id-keyed map exists only to tell "absent" apart from "absent, but a stale copy
under an older name is sitting there", and it stays unexported so nothing can
reach for it as an answer. That distinction is the bug this rewrite exists to
remove, so it is enforced by structure rather than by comment.
Names are checked for path containment before use. Storing them verbatim means
a filename chosen by whoever uploaded the file can contain a separator or a
parent reference, and tdl never had to care because its template rewrote those
away. Over-long names are refused for the same reason: the filesystem would
reject them at create time, and a file that can never be written would be
reported absent on every pass forever.
Objects are addressed by the path rclone knows them by, not by the basename
used for matching. The two differ once a remote has directory structure, and
deleting by basename would miss the object or remove a same-named one from the
root. Basenames appearing at more than one path make the snapshot ambiguous, so
they are reported rather than silently resolved.
Indexing runs at full depth with filters cleared. Inheriting RCLONE_MAX_DEPTH
or RCLONE_EXCLUDE would not fail, it would quietly report archived files as
absent and fetch them all again.
Deleting stale copies stays opt-in and confirmed; a non-interactive stdin
declines rather than proceeding. Filenames are quoted wherever they are
printed, so an embedded escape cannot redraw the list an operator approves.
Verified against the live remote: identical to verify-export.sh on the same
state — 18155 expected, 15548 present, 2607 absent, ids 9857-18013, exit 1.
First slice of replacing the three-script pipeline with one process. The
scripts coordinate tdl and rclone as separate programs, so everything
expensive in them exists to work around the fact that neither can see the
other's state. A single process does not need that machinery.
This slice covers only the foundations: open the session tdl login already
wrote, resolve an rclone destination, and report on both via a doctor
command. Downloading, uploading and verification follow.
The session store is shared with the tdl CLI rather than copied, so the two
cannot run against one namespace at the same time; -n selects another.
AppID and AppHash are read from the store rather than hardcoded, because a
session is bound to the application that created it and tdl records which
one it used.
No middlewares are passed to tclient.New, which already prepends its own
defaults; the DC pool gets them instead, since gotd applies a client's
middlewares only to direct invocations and not to pooled connections.
The rclone config is loaded up front because the lazy path calls os.Exit on
a config it cannot read, which would bypass every defer and exit with the
code this tool reserves for an incomplete run.
Exit codes follow the shell pipeline: 0 ok, 1 incomplete, 2 usage, 3 remote
failure, 130 SIGINT, 143 SIGTERM. Cancellation is checked explicitly after
the Telegram client returns, because gotd reports an interrupted run as
success and a driver would read that as a finished archive.
tiennm99