diff --git a/README.md b/README.md index 9bd0276..85fad4f 100644 --- a/README.md +++ b/README.md @@ -1 +1,274 @@ -# telegram-exporter \ No newline at end of file +# telegram-exporter + +`tg-export` downloads **all media from a Telegram group** to local disk, one +folder per logical post (albums collapsed), with a `messages.jsonl` sidecar that +links every file back to its message. Resumable, filterable, with a dry-run size +estimate. + +```bash +export TG_API_ID=... TG_API_HASH=... + +tg-export --group @mygroup --out ./exports --dry-run # what would it fetch? +tg-export --group @mygroup --out ./exports # fetch it +``` + +Kill it at any point and re-run the same command; it resumes without +re-downloading and without gaps. + +## Why the Bot API cannot do this + +This is the first question most people ask, so: a bot cannot read a group's +message history at all, and the Bot API caps file downloads at 20 MB. Both limits +are structural, not configuration. Exporting history therefore requires MTProto +with a **user account**, which is what Telethon provides. + +## Install + +Python 3.12+. + +```bash +python3 -m venv .venv +.venv/bin/pip install -r requirements.txt -e . +.venv/bin/tg-export --help +``` + +`requirements.txt` pins the exact Telethon version the API probes were verified +against. Telethon 2.0 is alpha with a renamed API surface, hence the `<2` bound. + +`cryptg` is **optional and not installed**. It is a native extension sitting in +Telethon's AES path, so it sees the auth key and every plaintext byte. It ships an +aarch64 wheel (no build step), and the reason to trust it *if* you adopt it is +that the Telethon author maintains it — not that it is "CPU-side only". Do not add +a supply-chain root for a throughput problem you have not measured. + +## Credentials + +Create an app at → API development tools. + +| Credential | Source | Why | +|---|---|---| +| `api_id`, `api_hash` | env `TG_API_ID` / `TG_API_HASH` only | needed every run; revocable | +| phone | prompt; `TG_PHONE` optional | PII, not a secret | +| login code | **prompt only** | single-use, 5-minute TTL | +| 2FA password | **`getpass` only** — no env, no file, no flag | once per session lifetime | + +`.env` **may** contain `TG_API_ID`, `TG_API_HASH`, `TG_PHONE`. It **must not** +contain the 2FA password, a login code, or session data. The app does not read +`.env` — your shell already does that: + +```bash +set -a; . ./.env; set +a +``` + +### First run and the session file + +The first run prompts for your phone, the login code Telegram sends you, and your +2FA password if you have one. Later runs are non-interactive. + +The session file defaults to +`${XDG_STATE_HOME:-~/.local/state}/tg-export/default.session` — **outside the +repo**, so the safe location needs no opt-in. It and every SQLite sibling are mode +`0600`, enforced by a `umask(0o077)` set before anything can create a file, plus a +post-login assertion. A `chmod` after connecting would be too late: SQLite writes +the auth key *during* login. + +If you point `--session` or `--out` inside a git repo at a path that is not +gitignored, the run refuses with an explanation. Deleting a session file does +**not** revoke it server-side — use Telegram → Settings → Devices for that. + +## Output layout + +``` +exports/g-1001234567890/ <- directory name is the chat id, never the group title + title.txt <- the human-readable name lives here, as data + 1042/ <- one album (shared grouped_id); folder = first member id + 1042_photo.jpg <- filename keyed on message id, not a positional index + 1043_photo.jpg + 1044_video.mp4 + 1047/ <- a standalone message + 1047_document.pdf + messages.jsonl + .export-state.json + .export.lock +``` + +The directory is `g` and not the group title because the title is +server-supplied and editable by any admin. A title of `..` or `a/b` would be a +path-traversal vector that no filename sanitizer could catch, since the escape +happens in a parent component. Naming the directory after an int makes that +structurally impossible — and also means a mid-export group rename cannot orphan +your download. The cost is that `ls exports/` shows numbers. + +Filenames are keyed on the **message id** rather than a position in the album, +because a positional index shifts when a member message is deleted between runs, +silently orphaning files already on disk. + +## Resume semantics + +`.export-state.json` holds one cursor, and it means: + +> every post with `max_message_id <= cursor_id` has been handled **under these +> filters**. + +Consequences worth knowing: + +- The cursor is written only after a post is completely handled, and never from an + error path. It can never point past in-flight work. +- Filters are stored verbatim. Changing them refuses the run and prints exactly + what changed, because the cursor's meaning depends on them: + + ``` + filter mismatch: + types: ['photo', 'video'] -> ['video'] + max_size: 100000000 -> (none) + re-run with the original filters, or --reset-state to start over + ``` + +- `--reset-state` discards the cursor and rotates `messages.jsonl` to + `messages-.jsonl`. It does **not** delete downloaded files — + dedupe reclaims them on the next run, and deleting 30 GB to change a filter + would be hostile. +- `--limit N` is *not* part of the filter set. It changes when we stop, not what a + post contains, so a `--limit 20` test run is a genuine resumable partial export + that does not poison later full runs. +- `completed_at` is set only when a run exhausts the history cleanly. It is the + only way to answer "did my 20-hour export actually finish?". +- **A file that fails is recorded and skipped, and the cursor moves past it.** The + run continues and the exit summary names the count (`N files failed (see + messages.jsonl)`). Media that was deleted or expired is genuinely gone, but a + file that exhausted its retries is not re-attempted by a later resume — grep + `errors` in `messages.jsonl` to find them, and use `--reset-state` to re-drive + that range if it matters. + +## `messages.jsonl` + +```json +{"message_id":1043,"post_id":1042,"grouped_id":88,"date":"2026-07-04T12:00:00+00:00", + "sender_id":777,"sender_name":"Alice","caption":"beach trip","reply_to":1001, + "files":[{"path":"1042/1043_photo.jpg","size":1234567,"name":"photo.jpg", + "mime":"image/jpeg","kind":"photo","declared_size":1234999}], + "errors":[]} +``` + +**Each record is an append-only event: "the files written for this message in this +run."** So the correct way to read the sidecar is the **union of all records for a +`message_id`**, never last-write-wins — after a filter change, the most recent +record deliberately describes a narrower slice than what is on disk. + +- `size` is bytes on disk; `declared_size` appears only when the server's number + differs (see the photo note below). +- Paths are relative to the export root. +- `sender_name` is filled in only when the sender was already cached. Resolving it + otherwise would mean one extra RPC per message — 100k RPCs and a flood ban on a + large group, in exchange for a display string. +- A partial final line from a host crash is truncated and warned about at startup. +- With `--include-text`, text-only messages get the same record shape with + `"files": []`. Distinguish media from context by `files` being empty, not by a + record type. + +## Flags + +``` +tg-export --group --out DIR + [--dry-run] [--since YYYY-MM-DD] [--until YYYY-MM-DD] + [--types photo,video,document,audio,voice] [--max-size 100MB] + [--include-text] [--limit N] [--session PATH] [--reset-state] + [--min-free 2GiB] [--max-flood-wait SECONDS] [--verbose] +``` + +- `--include-text` is off by default. When set, text-only messages are recorded in + the sidecar with an empty `files[]` so captions and replies keep their + surrounding conversation. They never create a folder and never download + anything. It **is** part of the stored filter set — it changes what a post + contains — so toggling it requires `--reset-state`. +- `--verbose` raises the log level for *this tool only*. The `telethon` logger is + pinned at `WARNING` unconditionally, because its request logging carries + `api_id`, `phone_number` and `phone_code_hash`. There is deliberately no + `--debug-telethon`. +- A link preview is not media: a message containing a URL is not treated as an + attachment, so preview thumbnails are never downloaded. + +## Flood waits have no ceiling by default + +Telegram throttles by telling the client to wait, and those waits escalate from +seconds to hours if you keep pushing. By default `tg-export` **sleeps as long as +Telegram demands**, so an unattended export completes rather than failing fast. + +Every wait is logged at `WARNING` with its duration **and its computed wake +time**: + +``` +2026-08-11 07:42:11 WARNING flood wait 3600s - sleeping until 2026-08-11T08:42:11+00:00 +``` + +A long sleep is a sleep, not a hang. Read the log before assuming otherwise. Pass +`--max-flood-wait SECONDS` to exit cleanly and resumably (code 6) instead. + +## Why downloads are sequential + +Telethon does not parallelize a single transfer, and running several at once +mainly accelerates flood-wait escalation — a 7-second wait becomes a 4-hour +lockout. There is deliberately no `--workers` flag, and an AST test in the suite +fails if `gather`, `create_task`, `to_thread` or a thread pool ever appears in +`src/`. + +## Disk space + +`--dry-run` reports remaining posts, files and bytes by media type against free +space, and exits 2 if it will not fit. + +**That verdict is advisory.** The enforcement is a per-file check before each +download, comparing free space against *this file's* size plus the `--min-free` +reserve. So a `SHORT` verdict does not mean the run is unsafe — it means the run +will stop partway with a clean, resumable cursor. + +Photo totals are labelled approximate because a photo's declared `.size` +describes the size variant Telethon selected. Documents, video and audio carry an +exact size. Media reporting no size at all is downloaded anyway and counted on its +own `unknown size` line rather than as zero, so the estimate never hides an +unbounded error. + +`--dry-run` reads the cursor and measures from where a real run would start, so +`tg-export --dry-run && tg-export` works on a partially completed export. It takes +no lock, creates no directories, and mutates nothing — you can run it while a real +export is in progress. + +## Concurrent runs + +One run per export root. A second run against the same `--out` exits 3 and names +the holder's pid, host and start time. `flock` releases on process death, `kill -9` +included, so there is no stale lock to clean up. + +Exporting two *different* groups at the same time works, but each needs its own +`--session` as well as its own `--out`: the session file is a SQLite database and +is locked too, because two Telethon clients sharing one session is a corruption +hazard Telethon itself warns about. + +`--dry-run` takes no lock at all, so it can always be run against an export that +is currently in progress. + +## Exit codes + +| Code | Meaning | +|---|---| +| 0 | success | +| 1 | unexpected error, or a bad argument | +| 2 | `--dry-run` says it will not fit | +| 3 | disk exhausted, or another run holds the lock | +| 4 | session invalidated — re-login | +| 5 | lost access to the group | +| 6 | flood wait exceeded `--max-flood-wait` (only when you set it) | +| 7 | filter or chat mismatch against the existing state file | +| 130 | interrupted — re-run to resume | + +## Development + +```bash +.venv/bin/pip install -r requirements.txt -e ".[dev]" +.venv/bin/pytest +``` + +The whole suite runs offline against synthetic message stubs — no credentials, no +network. Note that `messages.jsonl` contains message text and sender names from +other people; the export tree is gitignored and the runtime check refuses an +un-ignored path inside a repo, but check `git status` before any commit.