diff --git a/README.md b/README.md index 3025548..b22174f 100644 --- a/README.md +++ b/README.md @@ -66,7 +66,7 @@ and warnings go to stderr; press Ctrl-C at any point and it stops cleanly. | `-f FILE` | Export JSON to download from (default `export-.json` with `-c`, else `export.json`) | | `-d DIR` | Staging directory (default `./staging`) | | `-i SECONDS` | Seconds between rclone sweeps (default `60`) | -| `-a AGE` | rclone `--min-age`, a second guard against moving files still being written (default `2m`) | +| `-a AGE` | rclone `--min-age`, a second guard against moving files still being written (default `45s`) | | `-m SIZE` | Cap the staging directory at `SIZE` (`K`/`M`/`G`/`T`, binary), e.g. `40G`. Unset means no cap (see below) | | `-h` | Help | @@ -120,6 +120,28 @@ side is tunable without touching the script: RCLONE_TRANSFERS=8 RCLONE_BWLIMIT=20M ./run.sh -r s3:bucket/tg -c @mygroup ``` +`run.sh` sets two of those itself, and only when the caller has not: + +| Variable | Default here | rclone's own default | Why | +|----------|--------------|----------------------|-----| +| `RCLONE_TRANSFERS` | `2` | `4` | Backends that commit an upload as a server-side async task queue those tasks; less parallelism keeps the queue short | +| `RCLONE_LOW_LEVEL_RETRIES` | `20` | `10` | Each retry re-polls a pending task, so a slow commit is waited out instead of failing the transfer | + +Both exist because of one failure mode. On pikpak an upload finishes in two +phases — rclone sends the bytes, then a server-side task must reach +`PHASE_TYPE_COMPLETE`. rclone waits 500 ms and then polls, giving up after +`--low-level-retries` attempts with: + +``` +ERROR : : Failed to copy: can't verify the task is completed: ... Phase:"PHASE_TYPE_PENDING" +``` + +Nothing is lost when that happens — the message is followed by `Not deleting +source as copy failed`, the file stays in staging and the next sweep retries +it. But it wastes the upload, and it counts against a `-m` cap, since a file +that keeps failing can never be drained. Raise the retries further if you still +see it. + ### Capping the staging directory Without `-m`, staging grows whenever tdl downloads faster than rclone uploads, @@ -236,9 +258,9 @@ subset, and repeat. | `-c CHAT` | Chat to export metadata for on the first pass | | `-f FILE` | Export JSON (default `export-.json`) | | `-d DIR` | Staging directory (default `./staging`) | -| `-i SECONDS` | rclone sweep interval (default `300`) | +| `-i SECONDS` | rclone sweep interval (default `120`) | | `-m SIZE` | Staging cap passed through to `run.sh`, e.g. `40G` | -| `-p N` | Maximum passes (default `20`) | +| `-p N` | Maximum passes (default `30`) | | `-q GIB` | Stop if remote free space falls below this (default `5`) | It stops when the verifier reports complete (exit `0`), when a pass fetches diff --git a/export-until-complete.sh b/export-until-complete.sh index 0c6eeae..3095f32 100755 --- a/export-until-complete.sh +++ b/export-until-complete.sh @@ -13,15 +13,15 @@ # -c CHAT chat id/username, used for the initial metadata export # -f FILE export JSON (default export-.json) # -d DIR staging directory (default ./staging) -# -i SECONDS rclone sweep interval (default 300) -# -p N maximum passes (default 20) +# -i SECONDS rclone sweep interval (default 120) +# -p N maximum passes (default 30) # -m SIZE cap the staging directory at SIZE, e.g. 40G (default: no cap) # -q GIB stop if remote free space falls below this (default 5) set -euo pipefail -remote='' chat='' export_file='' staging='./staging' interval=300 max_staging='' -max_passes=20 min_free_gib=5 +remote='' chat='' export_file='' staging='./staging' interval=120 max_staging='' +max_passes=30 min_free_gib=5 while getopts ':r:c:f:d:i:p:q:m:h' o; do case $o in r) remote=$OPTARG ;; c) chat=$OPTARG ;; f) export_file=$OPTARG ;; diff --git a/run.sh b/run.sh index 17d4721..b552068 100755 --- a/run.sh +++ b/run.sh @@ -27,7 +27,7 @@ chat='' staging='./staging' remote='' interval=60 -min_age='2m' +min_age='45s' max_staging='' # empty: staging grows as fast as tdl fills it max_sync_failures=5 cap_check_interval=10 # seconds between -m checks, independent of -i @@ -132,6 +132,15 @@ if [[ -z $export_file ]]; then fi fi +# pikpak-style backends finish an upload as a server-side async task, and +# rclone abandons a still-pending one once --low-level-retries polls run out, +# failing a transfer that would have succeeded. Fewer parallel transfers keep +# that queue short and more retries wait it out. These are rclone's own +# environment variables, so whatever the caller already exported wins. +: "${RCLONE_TRANSFERS:=2}" +: "${RCLONE_LOW_LEVEL_RETRIES:=20}" +export RCLONE_TRANSFERS RCLONE_LOW_LEVEL_RETRIES + for tool in tdl rclone; do command -v "$tool" >/dev/null || die "$tool is not installed or not on PATH" done