From f77f9c4c1a56bddf622bd9d7a9f90706a7241f16 Mon Sep 17 00:00:00 2001 From: tiennm99 Date: Fri, 4 Sep 2026 10:14:09 +0700 Subject: [PATCH] feat: cap staging size and report progress on end-of-run sweeps Add -m SIZE to run.sh, passed through by export-until-complete.sh, so the staging directory cannot outgrow the local disk when tdl downloads faster than rclone uploads. Staging is measured every 10 seconds, independently of the sweep interval; at the cap tdl is suspended with SIGSTOP and rclone sweeps until staging is back under it, then tdl resumes and --continue picks its .tmp files back up. The drain loops while it frees space and gives up once only in-flight .tmp files remain, which no sweep can move, so a cap set below what the concurrent downloads hold warns instead of stalling. The failure streak still counts only sweeps that actually ran, so a quiet cap check cannot clear it. The sweeps that run once at the end now report progress rather than going quiet for minutes: rclone's redrawn bar on a terminal, one-line stats every 30s when output is redirected. Stats are logged at INFO, so only the stats level is raised to NOTICE, avoiding the line-per-file output -v would add. Also send SIGCONT before SIGTERM when stopping tdl, since a suspended process never sees the term. --- README.md | 52 +++++++++++++++ export-until-complete.sh | 12 ++-- run.sh | 137 ++++++++++++++++++++++++++++++++++----- 3 files changed, 180 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index 3a555f5..3025548 100644 --- a/README.md +++ b/README.md @@ -67,6 +67,7 @@ and warnings go to stderr; press Ctrl-C at any point and it stops cleanly. | `-d DIR` | Staging directory (default `./staging`) | | `-i SECONDS` | Seconds between rclone sweeps (default `60`) | | `-a AGE` | rclone `--min-age`, a second guard against moving files still being written (default `2m`) | +| `-m SIZE` | Cap the staging directory at `SIZE` (`K`/`M`/`G`/`T`, binary), e.g. `40G`. Unset means no cap (see below) | | `-h` | Help | ### Identifying the chat @@ -119,6 +120,29 @@ side is tunable without touching the script: RCLONE_TRANSFERS=8 RCLONE_BWLIMIT=20M ./run.sh -r s3:bucket/tg -c @mygroup ``` +### Capping the staging directory + +Without `-m`, staging grows whenever tdl downloads faster than rclone uploads, +which on a fast connection and a slow remote can mean tens of GB between +sweeps. `-m` puts a ceiling on it: + +```bash +./run.sh -r s3:bucket/tg -c @mygroup -m 40G +``` + +Staging size is checked every 10 seconds, independently of `-i`. When it +reaches the cap, tdl is suspended with `SIGSTOP` and rclone sweeps until +staging is back under it, then tdl is resumed — it reconnects on its own and +`--continue` picks its `.tmp` files back up. Because the checks are periodic, +the cap is a high-water mark rather than a hard limit: staging can overshoot by +up to ten seconds of download throughput before the gate closes. + +Only finished files can be drained, so the cap has to exceed what the +concurrent downloads hold — at most `-l` times 2 GB (4 GB from premium +uploaders). With the default `-l 2` anything from ~10 GB up is safe; below +that, the drain cannot clear the cap and the run logs a warning on every check +instead of throttling. + ### Exporting a subset Generate the JSON yourself when you want a narrower export, then point `-f` at @@ -132,6 +156,30 @@ tdl chat export -c @mygroup -T id -i 1000,5000 --all --with-content -o part.json `tdl chat export` takes `-T time|id|last` with `-i` as the range, and `-f` as an expression filter over message fields (`-f -` lists the available fields). +## Sweep output + +The periodic sweeps are silent — they run every `-i` seconds alongside tdl's own +output, and narrating each one would drown it. The sweeps that run **once at the +end** do report progress, since they can move the whole staging directory with +nothing else on screen: + +- the exit sweep on Ctrl-C, `SIGTERM`, or a tdl failure (`sweeping completed + files before exit`); +- the final sweep after tdl finishes successfully. + +On a terminal that is rclone's redrawn `--progress` bar. When output is +redirected to a log it becomes a one-line stats summary every 30s +(`--stats 30s --stats-one-line --stats-log-level NOTICE`) — rclone logs stats at +INFO, so raising just the stats to NOTICE avoids the line-per-file spam that +`-v` would add. + +To show progress on every sweep instead, rclone reads its flags from the +environment: + +```bash +RCLONE_PROGRESS=true ./run.sh -r s3:bucket/tg -c @mygroup +``` + ## Resuming Re-run the same command. Both legs resume independently and nothing is @@ -189,6 +237,7 @@ subset, and repeat. | `-f FILE` | Export JSON (default `export-.json`) | | `-d DIR` | Staging directory (default `./staging`) | | `-i SECONDS` | rclone sweep interval (default `300`) | +| `-m SIZE` | Staging cap passed through to `run.sh`, e.g. `40G` | | `-p N` | Maximum passes (default `20`) | | `-q GIB` | Stop if remote free space falls below this (default `5`) | @@ -216,6 +265,9 @@ is redirected, so a log file stays readable without a flag. and keeps staging for the next attempt. - **A dead remote filling the disk.** Five consecutive rclone failures abort the run instead of letting staging grow unbounded. +- **A fast connection filling the disk.** With `-m`, tdl is suspended whenever + staging reaches the cap and resumed once rclone has drained it, so download + throughput cannot outrun the upload leg. - **Typos and bad credentials.** Before downloading anything, the remote must be present in `rclone listremotes` (skipped for connection strings) and the destination must be creatable, which proves both reachability and auth. diff --git a/export-until-complete.sh b/export-until-complete.sh index becff63..0c6eeae 100755 --- a/export-until-complete.sh +++ b/export-until-complete.sh @@ -15,19 +15,20 @@ # -d DIR staging directory (default ./staging) # -i SECONDS rclone sweep interval (default 300) # -p N maximum passes (default 20) +# -m SIZE cap the staging directory at SIZE, e.g. 40G (default: no cap) # -q GIB stop if remote free space falls below this (default 5) set -euo pipefail -remote='' chat='' export_file='' staging='./staging' interval=300 +remote='' chat='' export_file='' staging='./staging' interval=300 max_staging='' max_passes=20 min_free_gib=5 -while getopts ':r:c:f:d:i:p:q:h' o; do case $o in +while getopts ':r:c:f:d:i:p:q:m:h' o; do case $o in r) remote=$OPTARG ;; c) chat=$OPTARG ;; f) export_file=$OPTARG ;; d) staging=$OPTARG ;; i) interval=$OPTARG ;; p) max_passes=$OPTARG ;; - q) min_free_gib=$OPTARG ;; - h) sed -n '2,18p' "$0"; exit 0 ;; - *) echo "usage: $0 -r REMOTE:PATH -c CHAT [-f FILE] [-i SECS] [-p N] [-q GIB]" >&2; exit 2 ;; + q) min_free_gib=$OPTARG ;; m) max_staging=$OPTARG ;; + h) sed -n '2,19p' "$0"; exit 0 ;; + *) echo "usage: $0 -r REMOTE:PATH -c CHAT [-f FILE] [-i SECS] [-p N] [-q GIB] [-m SIZE]" >&2; exit 2 ;; esac; done log() { printf '\n=== %s [driver] %s ===\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*"; } @@ -97,6 +98,7 @@ PY rc=0 ./run.sh -r "$remote" -f gap.json -d "$staging" -i "$interval" \ + ${max_staging:+-m "$max_staging"} \ -- --group=false -t 4 -l 2 ${tdl_quiet[@]+"${tdl_quiet[@]}"} || rc=$? log "pass $pass finished (run.sh exit $rc)" diff --git a/run.sh b/run.sh index 233c7dc..17d4721 100755 --- a/run.sh +++ b/run.sh @@ -28,7 +28,9 @@ staging='./staging' remote='' interval=60 min_age='2m' +max_staging='' # empty: staging grows as fast as tdl fills it max_sync_failures=5 +cap_check_interval=10 # seconds between -m checks, independent of -i usage() { cat <&2; } die() { log "error: $*"; exit 2; } # usage or precondition fail() { log "error: $*"; exit 3; } # rclone / pipeline failure -while getopts ':f:c:d:r:i:a:h' opt; do +while getopts ':f:c:d:r:i:a:m:h' opt; do case $opt in f) export_file=$OPTARG ;; c) chat=$OPTARG ;; @@ -67,6 +72,7 @@ while getopts ':f:c:d:r:i:a:h' opt; do r) remote=$OPTARG ;; i) interval=$OPTARG ;; a) min_age=$OPTARG ;; + m) max_staging=$OPTARG ;; h) usage; exit 0 ;; :) die "option -$OPTARG requires an argument" ;; ?) die "unknown option -$OPTARG (try -h)" ;; @@ -81,6 +87,22 @@ tdl_extra=("$@") [[ $min_age =~ ^[0-9]+(\.[0-9]+)?(ms|s|m|h|d|w|M|y)?$ ]] \ || die "-a must be an rclone duration, e.g. 2m" +# Sizes are compared in KiB because that is the unit 'du -sk' reports, which is +# also the unit that matters here: allocated blocks, not apparent length. +max_staging_kib='' +if [[ -n $max_staging ]]; then + [[ $max_staging =~ ^[0-9]+[KkMmGgTt]?$ ]] \ + || die "-m must be a size with an optional K/M/G/T suffix, e.g. 40G" + num=${max_staging%[KkMmGgTt]} + case ${max_staging#"$num"} in + ''|K|k) max_staging_kib=$num ;; + M|m) max_staging_kib=$((num * 1024)) ;; + G|g) max_staging_kib=$((num * 1024 * 1024)) ;; + T|t) max_staging_kib=$((num * 1024 * 1024 * 1024)) ;; + esac + ((max_staging_kib > 0)) || die "-m must be greater than zero" +fi + # tdl resolves a numeric argument as an MTProto id and anything else through # gotd's resolver, which handles '@name', 'name' and t.me/tg:// links. Two forms # still need help: Bot API ids carry a '-100' prefix that MTProto does not use, @@ -146,24 +168,80 @@ tdl_pid='' tdl_rc=0 sweep_ok=0 +# The sweeps that run once at the end have a whole staging directory to move +# and no interleaved tdl output, so they report progress instead of going quiet +# for several minutes. rclone's redrawn bar is right on a terminal but turns a +# redirected log into control characters, so a log gets periodic one-line +# stats. Those are logged at INFO, which '-v' would enable at the cost of a +# line per file, hence raising the stats to NOTICE rather than the whole log. +sweep_progress=(--progress) +[[ -t 1 ]] || sweep_progress=(--stats 30s --stats-one-line --stats-log-level NOTICE) + # Move whatever is finished. Partial .tmp files are always excluded. $1 is an # optional --min-age guard; $2 enables --delete-empty-src-dirs, which is safe # only once tdl has stopped — rclone removing a directory between tdl's # MkdirAll and Create makes tdl fail, and it can take the staging root too, -# hence the mkdir afterwards. +# hence the mkdir afterwards. $3 turns on the progress reporting above. sweep() { local rc=0 local args=(--exclude "$TEMP_GLOB") if [[ -n ${1:-} ]]; then args+=(--min-age "$1"); fi if ((${2:-0})); then args+=(--delete-empty-src-dirs); fi + if ((${3:-0})); then args+=(${sweep_progress[@]+"${sweep_progress[@]}"}); fi rclone move "$staging" "$remote" "${args[@]}" || rc=$? mkdir -p "$staging" return $rc } +staging_kib() { du -sk "$staging" 2>/dev/null | cut -f1; } + +over_cap() { + local used + used=$(staging_kib) + [[ -n $used ]] && ((used >= max_staging_kib)) +} + +# Enforce -m. tdl renames a file only once it is complete, so staging holds +# finished files plus the in-flight '*.tmp' ones, and only the former can be +# drained. Suspending tdl stops it adding more while rclone empties the +# directory; SIGSTOP is safe because tdl reconnects on resume and --continue +# picks its .tmp files back up. The cap must therefore stay well above what the +# concurrent downloads hold, or draining could never clear it. +drain_to_cap() { + local used rc=0 + used=$(staging_kib) + log "staging at $((used / 1024)) MiB, at or over the $((max_staging_kib / 1024)) MiB cap; suspending tdl to drain" + + kill -STOP "$tdl_pid" 2>/dev/null || true + # Sweep until staging is back under the cap, since one rclone move need not + # get there: a slow remote or a per-file error can leave finished files + # behind. Stop as soon as a sweep frees nothing, which means all that is + # left is in-flight .tmp files that no sweep can ever move. + local before + while :; do + before=$used + # No --min-age: tdl is frozen, so every non-.tmp file is finished by + # construction and waiting out the guard would only prolong the pause. + sweep '' || { rc=$?; break; } + used=$(staging_kib) + [[ -n $used ]] || break + ((used >= max_staging_kib && used < before)) || break + done + kill -CONT "$tdl_pid" 2>/dev/null || true + + if ((rc == 0)) && [[ -n $used ]] && ((used >= max_staging_kib)); then + log "warning: staging is still $((used / 1024)) MiB after draining — the cap is below what the in-flight downloads hold; raise -m or lower tdl's -l/-t" + else + log "resumed tdl, staging now $((${used:-0} / 1024)) MiB" + fi + return $rc +} + cleanup() { if [[ -n $tdl_pid ]] && kill -0 "$tdl_pid" 2>/dev/null; then log "stopping tdl (pid $tdl_pid)" + # A suspended process never sees SIGTERM, so let it run first. + kill -CONT "$tdl_pid" 2>/dev/null || true kill -TERM "$tdl_pid" 2>/dev/null || true for _ in 1 2 3 4 5 6 7 8 9 10; do kill -0 "$tdl_pid" 2>/dev/null || break @@ -173,33 +251,60 @@ cleanup() { fi if ((sweep_ok)); then log 'sweeping completed files before exit' - sweep "$min_age" || log 'warning: final safety sweep failed; staging kept' + sweep "$min_age" 0 1 || log 'warning: final safety sweep failed; staging kept' fi } trap cleanup EXIT trap 'log "interrupted (SIGINT)"; exit 130' INT trap 'log "terminated (SIGTERM)"; exit 143' TERM -log "downloading into $staging, moving to $remote every ${interval}s" +log "downloading into $staging, moving to $remote every ${interval}s${max_staging_kib:+, capped at $((max_staging_kib / 1024)) MiB}" tdl dl -f "$export_file" -d "$staging" \ --takeout --group --skip-same --continue ${tdl_extra[@]+"${tdl_extra[@]}"} & tdl_pid=$! sweep_ok=1 failures=0 +waited=0 while kill -0 "$tdl_pid" 2>/dev/null; do - # sleep as a job so signals are handled without waiting out the interval - sleep "$interval" & + # A long sweep interval must not let staging blow past the cap in between, so + # sleep in slices and check the cap on each one. Every sleep is a job, so + # signals are handled without waiting the slice out. + slice=$((interval - waited)) + if [[ -n $max_staging_kib ]] && ((slice > cap_check_interval)); then + slice=$cap_check_interval + fi + sleep "$slice" & wait $! 2>/dev/null || true + waited=$((waited + slice)) - if sweep "$min_age"; then - failures=0 - else - failures=$((failures + 1)) - log "warning: rclone sweep failed ($failures/$max_sync_failures)" - if ((failures >= max_sync_failures)); then - sweep_ok=0 - fail "rclone failed $failures times in a row; stopping before staging fills the disk" + # Only a sweep that actually ran says anything about rclone's health, so the + # failure streak is judged on those alone and a quiet cap check never + # clears it. + rc=0 + swept=0 + if ((waited >= interval)); then + waited=0 + swept=1 + sweep "$min_age" || rc=$? + fi + if ((rc == 0)) && [[ -n $max_staging_kib ]] && kill -0 "$tdl_pid" 2>/dev/null; then + if over_cap; then + swept=1 + drain_to_cap || rc=$? + fi + fi + + if ((swept)); then + if ((rc == 0)); then + failures=0 + else + failures=$((failures + 1)) + log "warning: rclone sweep failed ($failures/$max_sync_failures)" + if ((failures >= max_sync_failures)); then + sweep_ok=0 + fail "rclone failed $failures times in a row; stopping before staging fills the disk" + fi fi fi done @@ -210,10 +315,10 @@ sweep_ok=0 if ((tdl_rc != 0)); then log "tdl exited $tdl_rc; staging kept at $staging — re-run to resume" - sweep "$min_age" || log 'warning: sweep after failure did not complete' + sweep "$min_age" 0 1 || log 'warning: sweep after failure did not complete' exit "$tdl_rc" fi log 'tdl finished; final sweep' -sweep '' 1 || fail "final sweep failed; files remain in $staging" +sweep '' 1 1 || fail "final sweep failed; files remain in $staging" log "done — everything moved to $remote"