Backends that finish an upload as a server-side task, pikpak among them,
leave it pending long enough that rclone abandons the transfer once
--low-level-retries polls run out. Default RCLONE_TRANSFERS to 2 to keep
that task queue short and RCLONE_LOW_LEVEL_RETRIES to 20 to wait a slow
commit out. Both use the :=default form, so an exported value still wins.
Shorten the --min-age default to 45s: excluding '*.tmp' is what actually
keeps an unfinished download from being uploaded, so the age guard only
needs to cover a rename racing a sweep, and two minutes of dwell made
staging larger than it had to be.
Sweep every 120s rather than 300s in the driver, which shrinks the backlog
each sweep has to move, and raise the pass limit to 30 to suit chats that
need more rounds than 20 to converge.
Add -m SIZE to run.sh, passed through by export-until-complete.sh, so the
staging directory cannot outgrow the local disk when tdl downloads faster
than rclone uploads. Staging is measured every 10 seconds, independently of
the sweep interval; at the cap tdl is suspended with SIGSTOP and rclone
sweeps until staging is back under it, then tdl resumes and --continue picks
its .tmp files back up. The drain loops while it frees space and gives up
once only in-flight .tmp files remain, which no sweep can move, so a cap set
below what the concurrent downloads hold warns instead of stalling. The
failure streak still counts only sweeps that actually ran, so a quiet cap
check cannot clear it.
The sweeps that run once at the end now report progress rather than going
quiet for minutes: rclone's redrawn bar on a terminal, one-line stats every
30s when output is redirected. Stats are logged at INFO, so only the stats
level is raised to NOTICE, avoiding the line-per-file output -v would add.
Also send SIGCONT before SIGTERM when stopping tdl, since a suspended
process never sees the term.
run.sh finishes when tdl finishes, which does not mean every file arrived: a
dropped session, a stalled remote or an interrupted pass leaves gaps that
nothing reported.
verify-export.sh rebuilds the filename tdl produces for each media message in
the export JSON and checks the remote and staging for it. Messages carrying no
media are skipped rather than counted as gaps. A zero-byte file counts as
missing because rclone overwrites a size-mismatched destination, so a retry
repairs it; files under 1 KiB are reported but not retried, since real media is
sometimes that small and retrying them would never terminate.
export-until-complete.sh loops that check with run.sh, narrowing the export to
the outstanding ids each pass. It stops when the verifier reports complete, when
a pass fetches nothing new, when the remote runs low on space, or on Ctrl-C.
tdl's progress bar is shown on a terminal and suppressed when output is
redirected, so logs stay readable.
Exports, staging and logs are now ignored: they hold real message ids, file
names and text from an account.
tiennm99