feat: cap staging size and report progress on end-of-run sweeps

Add -m SIZE to run.sh, passed through by export-until-complete.sh, so the
staging directory cannot outgrow the local disk when tdl downloads faster
than rclone uploads. Staging is measured every 10 seconds, independently of
the sweep interval; at the cap tdl is suspended with SIGSTOP and rclone
sweeps until staging is back under it, then tdl resumes and --continue picks
its .tmp files back up. The drain loops while it frees space and gives up
once only in-flight .tmp files remain, which no sweep can move, so a cap set
below what the concurrent downloads hold warns instead of stalling. The
failure streak still counts only sweeps that actually ran, so a quiet cap
check cannot clear it.

The sweeps that run once at the end now report progress rather than going
quiet for minutes: rclone's redrawn bar on a terminal, one-line stats every
30s when output is redirected. Stats are logged at INFO, so only the stats
level is raised to NOTICE, avoiding the line-per-file output -v would add.

Also send SIGCONT before SIGTERM when stopping tdl, since a suspended
process never sees the term.

README.md

@@ -67,6 +67,7 @@ and warnings go to stderr; press Ctrl-C at any point and it stops cleanly.
 | `-d DIR` | Staging directory (default `./staging`) |
 | `-i SECONDS` | Seconds between rclone sweeps (default `60`) |
 | `-a AGE` | rclone `--min-age`, a second guard against moving files still being written (default `2m`) |
+| `-m SIZE` | Cap the staging directory at `SIZE` (`K`/`M`/`G`/`T`, binary), e.g. `40G`. Unset means no cap (see below) |
 | `-h` | Help |
 
 ### Identifying the chat
@@ -119,6 +120,29 @@ side is tunable without touching the script:
 RCLONE_TRANSFERS=8 RCLONE_BWLIMIT=20M ./run.sh -r s3:bucket/tg -c @mygroup
 ```
 
+### Capping the staging directory
+
+Without `-m`, staging grows whenever tdl downloads faster than rclone uploads,
+which on a fast connection and a slow remote can mean tens of GB between
+sweeps. `-m` puts a ceiling on it:
+
+```bash
+./run.sh -r s3:bucket/tg -c @mygroup -m 40G
+```
+
+Staging size is checked every 10 seconds, independently of `-i`. When it
+reaches the cap, tdl is suspended with `SIGSTOP` and rclone sweeps until
+staging is back under it, then tdl is resumed — it reconnects on its own and
+`--continue` picks its `.tmp` files back up. Because the checks are periodic,
+the cap is a high-water mark rather than a hard limit: staging can overshoot by
+up to ten seconds of download throughput before the gate closes.
+
+Only finished files can be drained, so the cap has to exceed what the
+concurrent downloads hold — at most `-l` times 2 GB (4 GB from premium
+uploaders). With the default `-l 2` anything from ~10 GB up is safe; below
+that, the drain cannot clear the cap and the run logs a warning on every check
+instead of throttling.
+
 ### Exporting a subset
 
 Generate the JSON yourself when you want a narrower export, then point `-f` at
@@ -132,6 +156,30 @@ tdl chat export -c @mygroup -T id -i 1000,5000 --all --with-content -o part.json
 `tdl chat export` takes `-T time|id|last` with `-i` as the range, and `-f` as an
 expression filter over message fields (`-f -` lists the available fields).
 
+## Sweep output
+
+The periodic sweeps are silent — they run every `-i` seconds alongside tdl's own
+output, and narrating each one would drown it. The sweeps that run **once at the
+end** do report progress, since they can move the whole staging directory with
+nothing else on screen:
+
+- the exit sweep on Ctrl-C, `SIGTERM`, or a tdl failure (`sweeping completed
+  files before exit`);
+- the final sweep after tdl finishes successfully.
+
+On a terminal that is rclone's redrawn `--progress` bar. When output is
+redirected to a log it becomes a one-line stats summary every 30s
+(`--stats 30s --stats-one-line --stats-log-level NOTICE`) — rclone logs stats at
+INFO, so raising just the stats to NOTICE avoids the line-per-file spam that
+`-v` would add.
+
+To show progress on every sweep instead, rclone reads its flags from the
+environment:
+
+```bash
+RCLONE_PROGRESS=true ./run.sh -r s3:bucket/tg -c @mygroup
+```
+
 ## Resuming
 
 Re-run the same command. Both legs resume independently and nothing is
@@ -189,6 +237,7 @@ subset, and repeat.
 | `-f FILE` | Export JSON (default `export-<chat>.json`) |
 | `-d DIR` | Staging directory (default `./staging`) |
 | `-i SECONDS` | rclone sweep interval (default `300`) |
+| `-m SIZE` | Staging cap passed through to `run.sh`, e.g. `40G` |
 | `-p N` | Maximum passes (default `20`) |
 | `-q GIB` | Stop if remote free space falls below this (default `5`) |
 
@@ -216,6 +265,9 @@ is redirected, so a log file stays readable without a flag.
   and keeps staging for the next attempt.
 - **A dead remote filling the disk.** Five consecutive rclone failures abort the
   run instead of letting staging grow unbounded.
+- **A fast connection filling the disk.** With `-m`, tdl is suspended whenever
+  staging reaches the cap and resumed once rclone has drained it, so download
+  throughput cannot outrun the upload leg.
 - **Typos and bad credentials.** Before downloading anything, the remote must be
   present in `rclone listremotes` (skipped for connection strings) and the
   destination must be creatable, which proves both reachability and auth.

export-until-complete.sh

@@ -15,19 +15,20 @@
 #   -d DIR          staging directory (default ./staging)
 #   -i SECONDS      rclone sweep interval (default 300)
 #   -p N            maximum passes (default 20)
+#   -m SIZE         cap the staging directory at SIZE, e.g. 40G (default: no cap)
 #   -q GIB          stop if remote free space falls below this (default 5)
 
 set -euo pipefail
 
-remote='' chat='' export_file='' staging='./staging' interval=300
+remote='' chat='' export_file='' staging='./staging' interval=300 max_staging=''
 max_passes=20 min_free_gib=5
 
-while getopts ':r:c:f:d:i:p:q:h' o; do case $o in
+while getopts ':r:c:f:d:i:p:q:m:h' o; do case $o in
   r) remote=$OPTARG ;;   c) chat=$OPTARG ;;      f) export_file=$OPTARG ;;
   d) staging=$OPTARG ;;  i) interval=$OPTARG ;;  p) max_passes=$OPTARG ;;
-  q) min_free_gib=$OPTARG ;;
-  h) sed -n '2,18p' "$0"; exit 0 ;;
-  *) echo "usage: $0 -r REMOTE:PATH -c CHAT [-f FILE] [-i SECS] [-p N] [-q GIB]" >&2; exit 2 ;;
+  q) min_free_gib=$OPTARG ;; m) max_staging=$OPTARG ;;
+  h) sed -n '2,19p' "$0"; exit 0 ;;
+  *) echo "usage: $0 -r REMOTE:PATH -c CHAT [-f FILE] [-i SECS] [-p N] [-q GIB] [-m SIZE]" >&2; exit 2 ;;
 esac; done
 
 log() { printf '\n=== %s [driver] %s ===\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*"; }
@@ -97,6 +98,7 @@ PY
 
   rc=0
   ./run.sh -r "$remote" -f gap.json -d "$staging" -i "$interval" \
+    ${max_staging:+-m "$max_staging"} \
     -- --group=false -t 4 -l 2 ${tdl_quiet[@]+"${tdl_quiet[@]}"} || rc=$?
   log "pass $pass finished (run.sh exit $rc)"
 

run.sh

@@ -28,7 +28,9 @@ staging='./staging'
 remote=''
 interval=60
 min_age='2m'
+max_staging=''      # empty: staging grows as fast as tdl fills it
 max_sync_failures=5
+cap_check_interval=10   # seconds between -m checks, independent of -i
 
 usage() {
   cat <<USAGE
@@ -48,6 +50,9 @@ Options:
   -i SECONDS      seconds between rclone sweeps (default: $interval)
   -a AGE          rclone --min-age, a second guard against moving files still
                   being written (default: $min_age)
+  -m SIZE         cap the staging directory at SIZE (K/M/G/T, binary; e.g.
+                  40G). When staging reaches it, tdl is suspended until
+                  rclone has drained the finished files. Unset means no cap.
   -h              this help
 
 Everything after -- is appended to the 'tdl dl' command, e.g.
@@ -59,7 +64,7 @@ log() { printf '%s [%s] %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$PROG" "$*" >&2; }
 die() { log "error: $*"; exit 2; }   # usage or precondition
 fail() { log "error: $*"; exit 3; }  # rclone / pipeline failure
 
-while getopts ':f:c:d:r:i:a:h' opt; do
+while getopts ':f:c:d:r:i:a:m:h' opt; do
   case $opt in
     f) export_file=$OPTARG ;;
     c) chat=$OPTARG ;;
@@ -67,6 +72,7 @@ while getopts ':f:c:d:r:i:a:h' opt; do
     r) remote=$OPTARG ;;
     i) interval=$OPTARG ;;
     a) min_age=$OPTARG ;;
+    m) max_staging=$OPTARG ;;
     h) usage; exit 0 ;;
     :) die "option -$OPTARG requires an argument" ;;
     ?) die "unknown option -$OPTARG (try -h)" ;;
@@ -81,6 +87,22 @@ tdl_extra=("$@")
 [[ $min_age =~ ^[0-9]+(\.[0-9]+)?(ms|s|m|h|d|w|M|y)?$ ]] \
   || die "-a must be an rclone duration, e.g. 2m"
 
+# Sizes are compared in KiB because that is the unit 'du -sk' reports, which is
+# also the unit that matters here: allocated blocks, not apparent length.
+max_staging_kib=''
+if [[ -n $max_staging ]]; then
+  [[ $max_staging =~ ^[0-9]+[KkMmGgTt]?$ ]] \
+    || die "-m must be a size with an optional K/M/G/T suffix, e.g. 40G"
+  num=${max_staging%[KkMmGgTt]}
+  case ${max_staging#"$num"} in
+    ''|K|k) max_staging_kib=$num ;;
+    M|m) max_staging_kib=$((num * 1024)) ;;
+    G|g) max_staging_kib=$((num * 1024 * 1024)) ;;
+    T|t) max_staging_kib=$((num * 1024 * 1024 * 1024)) ;;
+  esac
+  ((max_staging_kib > 0)) || die "-m must be greater than zero"
+fi
+
 # tdl resolves a numeric argument as an MTProto id and anything else through
 # gotd's resolver, which handles '@name', 'name' and t.me/tg:// links. Two forms
 # still need help: Bot API ids carry a '-100' prefix that MTProto does not use,
@@ -146,24 +168,80 @@ tdl_pid=''
 tdl_rc=0
 sweep_ok=0
 
+# The sweeps that run once at the end have a whole staging directory to move
+# and no interleaved tdl output, so they report progress instead of going quiet
+# for several minutes. rclone's redrawn bar is right on a terminal but turns a
+# redirected log into control characters, so a log gets periodic one-line
+# stats. Those are logged at INFO, which '-v' would enable at the cost of a
+# line per file, hence raising the stats to NOTICE rather than the whole log.
+sweep_progress=(--progress)
+[[ -t 1 ]] || sweep_progress=(--stats 30s --stats-one-line --stats-log-level NOTICE)
+
 # Move whatever is finished. Partial .tmp files are always excluded. $1 is an
 # optional --min-age guard; $2 enables --delete-empty-src-dirs, which is safe
 # only once tdl has stopped — rclone removing a directory between tdl's
 # MkdirAll and Create makes tdl fail, and it can take the staging root too,
-# hence the mkdir afterwards.
+# hence the mkdir afterwards. $3 turns on the progress reporting above.
 sweep() {
   local rc=0
   local args=(--exclude "$TEMP_GLOB")
   if [[ -n ${1:-} ]]; then args+=(--min-age "$1"); fi
   if ((${2:-0})); then args+=(--delete-empty-src-dirs); fi
+  if ((${3:-0})); then args+=(${sweep_progress[@]+"${sweep_progress[@]}"}); fi
   rclone move "$staging" "$remote" "${args[@]}" || rc=$?
   mkdir -p "$staging"
   return $rc
 }
 
+staging_kib() { du -sk "$staging" 2>/dev/null | cut -f1; }
+
+over_cap() {
+  local used
+  used=$(staging_kib)
+  [[ -n $used ]] && ((used >= max_staging_kib))
+}
+
+# Enforce -m. tdl renames a file only once it is complete, so staging holds
+# finished files plus the in-flight '*.tmp' ones, and only the former can be
+# drained. Suspending tdl stops it adding more while rclone empties the
+# directory; SIGSTOP is safe because tdl reconnects on resume and --continue
+# picks its .tmp files back up. The cap must therefore stay well above what the
+# concurrent downloads hold, or draining could never clear it.
+drain_to_cap() {
+  local used rc=0
+  used=$(staging_kib)
+  log "staging at $((used / 1024)) MiB, at or over the $((max_staging_kib / 1024)) MiB cap; suspending tdl to drain"
+
+  kill -STOP "$tdl_pid" 2>/dev/null || true
+  # Sweep until staging is back under the cap, since one rclone move need not
+  # get there: a slow remote or a per-file error can leave finished files
+  # behind. Stop as soon as a sweep frees nothing, which means all that is
+  # left is in-flight .tmp files that no sweep can ever move.
+  local before
+  while :; do
+    before=$used
+    # No --min-age: tdl is frozen, so every non-.tmp file is finished by
+    # construction and waiting out the guard would only prolong the pause.
+    sweep '' || { rc=$?; break; }
+    used=$(staging_kib)
+    [[ -n $used ]] || break
+    ((used >= max_staging_kib && used < before)) || break
+  done
+  kill -CONT "$tdl_pid" 2>/dev/null || true
+
+  if ((rc == 0)) && [[ -n $used ]] && ((used >= max_staging_kib)); then
+    log "warning: staging is still $((used / 1024)) MiB after draining — the cap is below what the in-flight downloads hold; raise -m or lower tdl's -l/-t"
+  else
+    log "resumed tdl, staging now $((${used:-0} / 1024)) MiB"
+  fi
+  return $rc
+}
+
 cleanup() {
   if [[ -n $tdl_pid ]] && kill -0 "$tdl_pid" 2>/dev/null; then
     log "stopping tdl (pid $tdl_pid)"
+    # A suspended process never sees SIGTERM, so let it run first.
+    kill -CONT "$tdl_pid" 2>/dev/null || true
     kill -TERM "$tdl_pid" 2>/dev/null || true
     for _ in 1 2 3 4 5 6 7 8 9 10; do
       kill -0 "$tdl_pid" 2>/dev/null || break
@@ -173,33 +251,60 @@ cleanup() {
   fi
   if ((sweep_ok)); then
     log 'sweeping completed files before exit'
-    sweep "$min_age" || log 'warning: final safety sweep failed; staging kept'
+    sweep "$min_age" 0 1 || log 'warning: final safety sweep failed; staging kept'
   fi
 }
 trap cleanup EXIT
 trap 'log "interrupted (SIGINT)"; exit 130' INT
 trap 'log "terminated (SIGTERM)"; exit 143' TERM
 
-log "downloading into $staging, moving to $remote every ${interval}s"
+log "downloading into $staging, moving to $remote every ${interval}s${max_staging_kib:+, capped at $((max_staging_kib / 1024)) MiB}"
 tdl dl -f "$export_file" -d "$staging" \
   --takeout --group --skip-same --continue ${tdl_extra[@]+"${tdl_extra[@]}"} &
 tdl_pid=$!
 sweep_ok=1
 
 failures=0
+waited=0
 while kill -0 "$tdl_pid" 2>/dev/null; do
-  # sleep as a job so signals are handled without waiting out the interval
-  sleep "$interval" &
+  # A long sweep interval must not let staging blow past the cap in between, so
+  # sleep in slices and check the cap on each one. Every sleep is a job, so
+  # signals are handled without waiting the slice out.
+  slice=$((interval - waited))
+  if [[ -n $max_staging_kib ]] && ((slice > cap_check_interval)); then
+    slice=$cap_check_interval
+  fi
+  sleep "$slice" &
   wait $! 2>/dev/null || true
+  waited=$((waited + slice))
 
-  if sweep "$min_age"; then
-    failures=0
-  else
-    failures=$((failures + 1))
-    log "warning: rclone sweep failed ($failures/$max_sync_failures)"
-    if ((failures >= max_sync_failures)); then
-      sweep_ok=0
-      fail "rclone failed $failures times in a row; stopping before staging fills the disk"
+  # Only a sweep that actually ran says anything about rclone's health, so the
+  # failure streak is judged on those alone and a quiet cap check never
+  # clears it.
+  rc=0
+  swept=0
+  if ((waited >= interval)); then
+    waited=0
+    swept=1
+    sweep "$min_age" || rc=$?
+  fi
+  if ((rc == 0)) && [[ -n $max_staging_kib ]] && kill -0 "$tdl_pid" 2>/dev/null; then
+    if over_cap; then
+      swept=1
+      drain_to_cap || rc=$?
+    fi
+  fi
+
+  if ((swept)); then
+    if ((rc == 0)); then
+      failures=0
+    else
+      failures=$((failures + 1))
+      log "warning: rclone sweep failed ($failures/$max_sync_failures)"
+      if ((failures >= max_sync_failures)); then
+        sweep_ok=0
+        fail "rclone failed $failures times in a row; stopping before staging fills the disk"
+      fi
     fi
   fi
 done
@@ -210,10 +315,10 @@ sweep_ok=0
 
 if ((tdl_rc != 0)); then
   log "tdl exited $tdl_rc; staging kept at $staging — re-run to resume"
-  sweep "$min_age" || log 'warning: sweep after failure did not complete'
+  sweep "$min_age" 0 1 || log 'warning: sweep after failure did not complete'
   exit "$tdl_rc"
 fi
 
 log 'tdl finished; final sweep'
-sweep '' 1 || fail "final sweep failed; files remain in $staging"
+sweep '' 1 1 || fail "final sweep failed; files remain in $staging"
 log "done — everything moved to $remote"