mirror of
https://github.com/tiennm99/telegram-exporter.git
synced 2026-10-14 00:16:59 +00:00
tiennm99
3bc4fd846b
The README now covers only what the tool does and how to run it: setup, options, tdl flag pass-through, subset exports, resume semantics, the failure modes it guards against, and exit codes. Flags are documented from tdl's own source rather than assumed: dl -f takes the exported JSON while dl -i/-e are file-extension filters, and the expression filter is chat export -f. Chats are addressed by id or domain.
125 lines
5.0 KiB
Markdown
125 lines
5.0 KiB
Markdown
# telegram-exporter
|
|
|
|
Export a Telegram chat's media to a **WebDAV** server, using far less local disk
|
|
than the chat's total size.
|
|
|
|
`run.sh` runs [tdl](https://github.com/iyear/tdl) and
|
|
[rclone](https://rclone.org/) as a rolling pipeline: tdl downloads into a small
|
|
staging directory while rclone concurrently moves finished files to WebDAV and
|
|
deletes the local copies. Local disk only ever holds the files in flight plus
|
|
one sync interval of throughput, so a multi-terabyte chat exports fine on a
|
|
small disk. Telegram caps a single file at 2 GB (4 GB from premium uploaders),
|
|
so a few dozen GB of staging covers the worst case regardless of chat size.
|
|
|
|
This is needed because **tdl can only write to a local directory** — it has no
|
|
rclone integration and no remote destination of any kind (`tdl dl -d` takes a
|
|
filesystem path; `tdl --storage` is its session database, not an output target).
|
|
tdl downloads over MTProto with a user account, so Bot API limits do not apply:
|
|
full history is readable and there is no 20 MB download cap.
|
|
|
|
## Requirements
|
|
|
|
- **tdl** — <https://docs.iyear.me/tdl/getting-started/installation/>
|
|
- **rclone** — <https://rclone.org/install/>
|
|
- **bash**. On Windows, run under WSL or Git Bash.
|
|
|
|
## Setup
|
|
|
|
Both steps are one-time.
|
|
|
|
```bash
|
|
# 1) log in to Telegram with your user account (phone + code + 2FA)
|
|
tdl login
|
|
|
|
# 2) create the WebDAV remote
|
|
rclone config create tg-webdav webdav \
|
|
url=https://dav.example.com/remote.php/dav/files/you \
|
|
vendor=other user=YOU pass=SECRET
|
|
|
|
rclone lsd tg-webdav: # verify it works
|
|
```
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
./run.sh -r tg-webdav:tg-export -c @mygroup
|
|
```
|
|
|
|
That is the whole flow. It exports the chat's message metadata to
|
|
`export.json`, then downloads and uploads concurrently until finished. Progress
|
|
and warnings go to stderr; press Ctrl-C at any point and it stops cleanly.
|
|
|
|
### Options
|
|
|
|
| Flag | Meaning |
|
|
|------|---------|
|
|
| `-r REMOTE:PATH` | **Required.** rclone destination, e.g. `tg-webdav:tg-export` |
|
|
| `-c CHAT` | Chat to export when the JSON does not exist yet: `@username` or a numeric chat id |
|
|
| `-f FILE` | Export JSON to download from (default `export.json`) |
|
|
| `-d DIR` | Staging directory (default `./staging`) |
|
|
| `-i SECONDS` | Seconds between rclone sweeps (default `60`) |
|
|
| `-a AGE` | rclone `--min-age`, a second guard against moving files still being written (default `2m`) |
|
|
| `-h` | Help |
|
|
|
|
Anything after `--` is passed straight to `tdl dl`:
|
|
|
|
```bash
|
|
./run.sh -r tg-webdav:tg-export -- -t 4 -l 1 # calmer parallelism, fewer flood waits
|
|
./run.sh -r tg-webdav:tg-export -- -i mp4,mkv # only these file extensions
|
|
./run.sh -r tg-webdav:tg-export -- -e jpg,png # skip these file extensions
|
|
```
|
|
|
|
tdl defaults to `-t 8 -l 4`, which is aggressive; lower it if you hit flood
|
|
waits on a large export.
|
|
|
|
### Exporting a subset
|
|
|
|
Generate the JSON yourself when you want a narrower export, then point `-f` at
|
|
it:
|
|
|
|
```bash
|
|
tdl chat export -c @mygroup -T id -i 1000,5000 --all --with-content -o part.json
|
|
./run.sh -r tg-webdav:tg-export -f part.json
|
|
```
|
|
|
|
`tdl chat export` takes `-T time|id|last` with `-i` as the range, and `-f` as an
|
|
expression filter over message fields (`-f -` lists the available fields).
|
|
|
|
## Resuming
|
|
|
|
Re-run the same command. Both legs resume independently and nothing is
|
|
downloaded or uploaded twice.
|
|
|
|
Keep the same `export.json` between runs: `--skip-same` compares against the
|
|
**staging** directory, which is empty once files have moved to WebDAV, so
|
|
cross-run deduplication rests on tdl's own `--continue` tracking. If you must
|
|
start from a fresh export, narrow it to the missing message-id range
|
|
(`-T id -i <last>,<max>`) rather than re-downloading everything.
|
|
|
|
## What it guards against
|
|
|
|
- **Partial uploads.** tdl writes `<name>.tmp` and renames on completion, so
|
|
every sweep excludes `*.tmp`. Age alone is not a completion signal: a download
|
|
stalled by a flood wait stops touching its `.tmp`, which would then be
|
|
uploaded half-written and lose its resume point.
|
|
- **Directories vanishing under tdl.** `--delete-empty-src-dirs` runs only in
|
|
the final sweep, and the staging directory is recreated after every sweep.
|
|
Removing a directory under a running tdl makes it fail to create its next file.
|
|
- **Orphaned downloads.** tdl is stopped on exit, Ctrl-C, or `SIGTERM`, so no
|
|
download keeps running after the script is gone.
|
|
- **A failed run looking finished.** The unrestricted final sweep happens only
|
|
after tdl exits 0. An interrupted or crashed run gets the age-guarded sweep
|
|
and keeps staging for the next attempt.
|
|
- **A dead remote filling the disk.** Five consecutive rclone failures abort the
|
|
run instead of letting staging grow unbounded.
|
|
|
|
Exit codes: `0` success, `2` usage error, `3` rclone failure, `130`/`143`
|
|
interrupted, anything else is tdl's own exit code.
|
|
|
|
## Limits
|
|
|
|
- Streaming with no staging at all (piping download chunks straight into a
|
|
WebDAV `PUT`) is not possible with tdl and would require custom code.
|
|
- The script is bash; the two tools it drives are cross-platform, but Windows
|
|
needs WSL or Git Bash.
|