From 4ae19e32eca5b0f2427a0d10b93c3449513bf0fd Mon Sep 17 00:00:00 2001 From: tiennm99 Date: Tue, 25 Aug 2026 16:07:03 +0700 Subject: [PATCH] feat: target any rclone remote instead of only WebDAV The pipeline never depended on WebDAV; only the naming, docs and preflight did. Replace the WebDAV-root listing with a portable check: a named remote must appear in 'rclone listremotes', and creating the destination proves reachability and credentials. On-the-fly connection strings have no config entry, so the name check is skipped for them. Read the remote list into a variable rather than piping it into 'grep -q': grep exits on the first match and kills rclone with SIGPIPE, which pipefail reports as failure, intermittently rejecting a configured remote. Document that rclone's flags are all settable through their RCLONE_* environment variables, so the upload side is tunable without new options. --- README.md | 56 +++++++++++++++++++++++++++++++++++++------------------ run.sh | 30 +++++++++++++++++++++-------- 2 files changed, 60 insertions(+), 26 deletions(-) diff --git a/README.md b/README.md index c0b1021..ea08cc5 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,14 @@ # telegram-exporter -Export a Telegram chat's media to a **WebDAV** server, using far less local disk -than the chat's total size. +Export a Telegram chat's media to **any rclone remote** — S3, Google Drive, +Dropbox, Backblaze B2, SFTP, WebDAV, or anything else rclone supports — using +far less local disk than the chat's total size. `run.sh` runs [tdl](https://github.com/iyear/tdl) and [rclone](https://rclone.org/) as a rolling pipeline: tdl downloads into a small -staging directory while rclone concurrently moves finished files to WebDAV and -deletes the local copies. Local disk only ever holds the files in flight plus -one sync interval of throughput, so a multi-terabyte chat exports fine on a +staging directory while rclone concurrently moves finished files to the remote +and deletes the local copies. Local disk only ever holds the files in flight +plus one sync interval of throughput, so a multi-terabyte chat exports fine on a small disk. Telegram caps a single file at 2 GB (4 GB from premium uploaders), so a few dozen GB of staging covers the worst case regardless of chat size. @@ -31,18 +32,25 @@ Both steps are one-time. # 1) log in to Telegram with your user account (phone + code + 2FA) tdl login -# 2) create the WebDAV remote -rclone config create tg-webdav webdav \ - url=https://dav.example.com/remote.php/dav/files/you \ +# 2) configure the destination. The interactive wizard covers every backend: +rclone config + +# ...or create one non-interactively, e.g. +rclone config create gdrive drive +rclone config create b2 b2 account=KEY_ID key=APP_KEY +rclone config create dav webdav url=https://dav.example.com/remote.php/dav/files/you \ vendor=other user=YOU pass=SECRET -rclone lsd tg-webdav: # verify it works +rclone listremotes # confirm the name you will pass to -r ``` +Any rclone remote form works, including on-the-fly connection strings +(`:webdav,url=https://...:/path`). + ## Usage ```bash -./run.sh -r tg-webdav:tg-export -c @mygroup +./run.sh -r gdrive:telegram/media -c @mygroup ``` That is the whole flow. It exports the chat's message metadata to @@ -53,7 +61,7 @@ and warnings go to stderr; press Ctrl-C at any point and it stops cleanly. | Flag | Meaning | |------|---------| -| `-r REMOTE:PATH` | **Required.** rclone destination, e.g. `tg-webdav:tg-export` | +| `-r REMOTE:PATH` | **Required.** rclone destination, e.g. `gdrive:telegram/media`, `s3:bucket/tg`, `dav:tg-export` | | `-c CHAT` | Chat to export when the JSON does not exist yet: `@username` or a numeric chat id | | `-f FILE` | Export JSON to download from (default `export.json`) | | `-d DIR` | Staging directory (default `./staging`) | @@ -64,14 +72,23 @@ and warnings go to stderr; press Ctrl-C at any point and it stops cleanly. Anything after `--` is passed straight to `tdl dl`: ```bash -./run.sh -r tg-webdav:tg-export -- -t 4 -l 1 # calmer parallelism, fewer flood waits -./run.sh -r tg-webdav:tg-export -- -i mp4,mkv # only these file extensions -./run.sh -r tg-webdav:tg-export -- -e jpg,png # skip these file extensions +./run.sh -r gdrive:telegram/media -- -t 4 -l 1 # calmer parallelism, fewer flood waits +./run.sh -r gdrive:telegram/media -- -i mp4,mkv # only these file extensions +./run.sh -r gdrive:telegram/media -- -e jpg,png # skip these file extensions ``` tdl defaults to `-t 8 -l 4`, which is aggressive; lower it if you hit flood waits on a large export. +### Tuning the upload + +rclone reads every one of its flags from an environment variable, so the upload +side is tunable without touching the script: + +```bash +RCLONE_TRANSFERS=8 RCLONE_BWLIMIT=20M ./run.sh -r s3:bucket/tg -c @mygroup +``` + ### Exporting a subset Generate the JSON yourself when you want a narrower export, then point `-f` at @@ -79,7 +96,7 @@ it: ```bash tdl chat export -c @mygroup -T id -i 1000,5000 --all --with-content -o part.json -./run.sh -r tg-webdav:tg-export -f part.json +./run.sh -r gdrive:telegram/media -f part.json ``` `tdl chat export` takes `-T time|id|last` with `-i` as the range, and `-f` as an @@ -91,7 +108,7 @@ Re-run the same command. Both legs resume independently and nothing is downloaded or uploaded twice. Keep the same `export.json` between runs: `--skip-same` compares against the -**staging** directory, which is empty once files have moved to WebDAV, so +**staging** directory, which is empty once files have moved to the remote, so cross-run deduplication rests on tdl's own `--continue` tracking. If you must start from a fresh export, narrow it to the missing message-id range (`-T id -i ,`) rather than re-downloading everything. @@ -112,13 +129,16 @@ start from a fresh export, narrow it to the missing message-id range and keeps staging for the next attempt. - **A dead remote filling the disk.** Five consecutive rclone failures abort the run instead of letting staging grow unbounded. +- **Typos and bad credentials.** Before downloading anything, the remote must be + present in `rclone listremotes` (skipped for connection strings) and the + destination must be creatable, which proves both reachability and auth. Exit codes: `0` success, `2` usage error, `3` rclone failure, `130`/`143` interrupted, anything else is tdl's own exit code. ## Limits -- Streaming with no staging at all (piping download chunks straight into a - WebDAV `PUT`) is not possible with tdl and would require custom code. +- Streaming with no staging at all (piping download chunks straight to the + remote) is not possible with tdl and would require custom code. - The script is bash; the two tools it drives are cross-platform, but Windows needs WSL or Git Bash. diff --git a/run.sh b/run.sh index 1c4c302..613d9f7 100755 --- a/run.sh +++ b/run.sh @@ -1,11 +1,12 @@ #!/usr/bin/env bash # # Rolling pipeline: tdl downloads Telegram media into a small staging directory -# while rclone concurrently moves finished files to a WebDAV remote. Local disk -# only ever holds the files in flight plus one sync interval of throughput, so a -# group larger than the local disk can still be exported. +# while rclone concurrently moves finished files to any rclone remote (S3, +# Google Drive, SFTP, WebDAV, B2, ...). Local disk only ever holds the files in +# flight plus one sync interval of throughput, so a chat larger than the local +# disk can still be exported. # -# See README.md ("Special case: exporting to WebDAV") for the background. +# See README.md for the background. # # Exit codes: 0 ok, 2 usage error, 3 rclone failure, 130/143 interrupted, # anything else is tdl's own exit code. @@ -34,7 +35,7 @@ usage() { Usage: $PROG -r REMOTE:PATH [options] [-- extra tdl dl args...] Required: - -r REMOTE:PATH rclone destination, e.g. tg-webdav:tg-export + -r REMOTE:PATH rclone destination, e.g. gdrive:telegram/media Options: -f FILE tdl export JSON (default: $export_file) @@ -46,7 +47,7 @@ Options: -h this help Everything after -- is appended to the 'tdl dl' command, e.g. - $PROG -r tg-webdav:tg-export -- -t 4 -l 1 + $PROG -r gdrive:telegram/media -- -t 4 -l 1 USAGE } @@ -80,8 +81,21 @@ for tool in tdl rclone; do command -v "$tool" >/dev/null || die "$tool is not installed or not on PATH" done -rclone lsd "${remote%%:*}:" >/dev/null 2>&1 \ - || die "rclone cannot reach remote '${remote%%:*}:' — check 'rclone config'" +# A named remote must exist in the config; a leading ':' means an on-the-fly +# connection string, which has no config entry to check. Listing the remote's +# root is not portable (some backends refuse it), so reachability and +# credentials are proven by creating the destination, which rclone would create +# on the first move anyway. +if [[ $remote != :* ]]; then + # Read the list into a variable first: piping it into 'grep -q' lets grep exit + # on the first match and kill rclone with SIGPIPE, which pipefail then reports + # as a failed pipeline -- rejecting a remote that is in fact configured. + remotes=$(rclone listremotes 2>/dev/null || true) + grep -qx -- "${remote%%:*}:" <<<"$remotes" \ + || die "rclone remote '${remote%%:*}:' is not configured — see 'rclone listremotes'" +fi +rclone mkdir "$remote" >/dev/null 2>&1 \ + || die "cannot reach '$remote' — check credentials and connectivity" # The export JSON only lists messages; it is cheap to keep and required for both # legs to stay resumable, so never regenerate it when it already exists.