# telegram-exporter `tg-export` downloads **all media from a Telegram group** to local disk, one folder per logical post (albums collapsed), with a `messages.jsonl` sidecar that links every file back to its message. Resumable, filterable, with a dry-run size estimate. ```bash export TG_API_ID=... TG_API_HASH=... tg-export --group @mygroup --out ./exports --dry-run # what would it fetch? tg-export --group @mygroup --out ./exports # fetch it ``` Kill it at any point and re-run the same command; it resumes without re-downloading and without gaps. ## Why the Bot API cannot do this This is the first question most people ask, so: a bot cannot read a group's message history at all, and the Bot API caps file downloads at 20 MB. Both limits are structural, not configuration. Exporting history therefore requires MTProto with a **user account**, which is what Telethon provides. ## Install Python 3.12+. ```bash python3 -m venv .venv .venv/bin/pip install -r requirements.txt -e . .venv/bin/tg-export --help ``` `requirements.txt` pins the exact Telethon version the API probes were verified against. Telethon 2.0 is alpha with a renamed API surface, hence the `<2` bound. `cryptg` is **optional and not installed**. It is a native extension sitting in Telethon's AES path, so it sees the auth key and every plaintext byte. It ships an aarch64 wheel (no build step), and the reason to trust it *if* you adopt it is that the Telethon author maintains it — not that it is "CPU-side only". Do not add a supply-chain root for a throughput problem you have not measured. ## Credentials Create an app at → API development tools. | Credential | Source | Why | |---|---|---| | `api_id`, `api_hash` | env `TG_API_ID` / `TG_API_HASH` only | needed every run; revocable | | phone | prompt; `TG_PHONE` optional | PII, not a secret | | login code | **prompt only** | single-use, 5-minute TTL | | 2FA password | **`getpass` only** — no env, no file, no flag | once per session lifetime | `.env` **may** contain `TG_API_ID`, `TG_API_HASH`, `TG_PHONE`. It **must not** contain the 2FA password, a login code, or session data. The app does not read `.env` — your shell already does that: ```bash set -a; . ./.env; set +a ``` ### First run and the session file The first run prompts for your phone, the login code Telegram sends you, and your 2FA password if you have one. Later runs are non-interactive. The session file defaults to `${XDG_STATE_HOME:-~/.local/state}/tg-export/default.session` — **outside the repo**, so the safe location needs no opt-in. It and every SQLite sibling are mode `0600`, enforced by a `umask(0o077)` set before anything can create a file, plus a post-login assertion. A `chmod` after connecting would be too late: SQLite writes the auth key *during* login. If you point `--session` or `--out` inside a git repo at a path that is not gitignored, the run refuses with an explanation. Deleting a session file does **not** revoke it server-side — use Telegram → Settings → Devices for that. ## Output layout ``` exports/g-1001234567890/ <- directory name is the chat id, never the group title title.txt <- the human-readable name lives here, as data 1042/ <- one album (shared grouped_id); folder = first member id 1042_photo.jpg <- filename keyed on message id, not a positional index 1043_photo.jpg 1044_video.mp4 1047/ <- a standalone message 1047_document.pdf messages.jsonl .export-state.json .export.lock ``` The directory is `g` and not the group title because the title is server-supplied and editable by any admin. A title of `..` or `a/b` would be a path-traversal vector that no filename sanitizer could catch, since the escape happens in a parent component. Naming the directory after an int makes that structurally impossible — and also means a mid-export group rename cannot orphan your download. The cost is that `ls exports/` shows numbers. Filenames are keyed on the **message id** rather than a position in the album, because a positional index shifts when a member message is deleted between runs, silently orphaning files already on disk. ## Resume semantics `.export-state.json` holds one cursor, and it means: > every post with `max_message_id <= cursor_id` has been handled **under these > filters**. Consequences worth knowing: - The cursor is written only after a post is completely handled, and never from an error path. It can never point past in-flight work. - Filters are stored verbatim. Changing them refuses the run and prints exactly what changed, because the cursor's meaning depends on them: ``` filter mismatch: types: ['photo', 'video'] -> ['video'] max_size: 100000000 -> (none) re-run with the original filters, or --reset-state to start over ``` - `--reset-state` discards the cursor and rotates `messages.jsonl` to `messages-.jsonl`. It does **not** delete downloaded files — dedupe reclaims them on the next run, and deleting 30 GB to change a filter would be hostile. - `--limit N` is *not* part of the filter set. It changes when we stop, not what a post contains, so a `--limit 20` test run is a genuine resumable partial export that does not poison later full runs. - `completed_at` is set only when a run exhausts the history cleanly. It is the only way to answer "did my 20-hour export actually finish?". - **A file that fails is recorded and skipped, and the cursor moves past it.** The run continues and the exit summary names the count (`N files failed (see messages.jsonl)`). Media that was deleted or expired is genuinely gone, but a file that exhausted its retries is not re-attempted by a later resume — grep `errors` in `messages.jsonl` to find them, and use `--reset-state` to re-drive that range if it matters. ## `messages.jsonl` ```json {"message_id":1043,"post_id":1042,"grouped_id":88,"date":"2026-07-04T12:00:00+00:00", "sender_id":777,"sender_name":"Alice","caption":"beach trip","reply_to":1001, "files":[{"path":"1042/1043_photo.jpg","size":1234567,"name":"photo.jpg", "mime":"image/jpeg","kind":"photo","declared_size":1234999}], "errors":[]} ``` **Each record is an append-only event: "the files written for this message in this run."** So the correct way to read the sidecar is the **union of all records for a `message_id`**, never last-write-wins — after a filter change, the most recent record deliberately describes a narrower slice than what is on disk. - `size` is bytes on disk; `declared_size` appears only when the server's number differs (see the photo note below). - Paths are relative to the export root. - `sender_name` is filled in only when the sender was already cached. Resolving it otherwise would mean one extra RPC per message — 100k RPCs and a flood ban on a large group, in exchange for a display string. - A partial final line from a host crash is truncated and warned about at startup. - With `--include-text`, text-only messages get the same record shape with `"files": []`. Distinguish media from context by `files` being empty, not by a record type. ## Flags ``` tg-export --group --out DIR [--dry-run] [--since YYYY-MM-DD] [--until YYYY-MM-DD] [--types photo,video,document,audio,voice] [--max-size 100MB] [--include-text] [--limit N] [--session PATH] [--reset-state] [--min-free 2GiB] [--max-flood-wait SECONDS] [--verbose] ``` - `--include-text` is off by default. When set, text-only messages are recorded in the sidecar with an empty `files[]` so captions and replies keep their surrounding conversation. They never create a folder and never download anything. It **is** part of the stored filter set — it changes what a post contains — so toggling it requires `--reset-state`. - `--verbose` raises the log level for *this tool only*. The `telethon` logger is pinned at `WARNING` unconditionally, because its request logging carries `api_id`, `phone_number` and `phone_code_hash`. There is deliberately no `--debug-telethon`. - A link preview is not media: a message containing a URL is not treated as an attachment, so preview thumbnails are never downloaded. ## Flood waits have no ceiling by default Telegram throttles by telling the client to wait, and those waits escalate from seconds to hours if you keep pushing. By default `tg-export` **sleeps as long as Telegram demands**, so an unattended export completes rather than failing fast. Every wait is logged at `WARNING` with its duration **and its computed wake time**: ``` 2026-08-11 07:42:11 WARNING flood wait 3600s - sleeping until 2026-08-11T08:42:11+00:00 ``` A long sleep is a sleep, not a hang. Read the log before assuming otherwise. Pass `--max-flood-wait SECONDS` to exit cleanly and resumably (code 6) instead. ## Why downloads are sequential Telethon does not parallelize a single transfer, and running several at once mainly accelerates flood-wait escalation — a 7-second wait becomes a 4-hour lockout. There is deliberately no `--workers` flag, and an AST test in the suite fails if `gather`, `create_task`, `to_thread` or a thread pool ever appears in `src/`. ## Disk space `--dry-run` reports remaining posts, files and bytes by media type against free space, and exits 2 if it will not fit. **That verdict is advisory.** The enforcement is a per-file check before each download, comparing free space against *this file's* size plus the `--min-free` reserve. So a `SHORT` verdict does not mean the run is unsafe — it means the run will stop partway with a clean, resumable cursor. Photo totals are labelled approximate because a photo's declared `.size` describes the size variant Telethon selected. Documents, video and audio carry an exact size. Media reporting no size at all is downloaded anyway and counted on its own `unknown size` line rather than as zero, so the estimate never hides an unbounded error. `--dry-run` reads the cursor and measures from where a real run would start, so `tg-export --dry-run && tg-export` works on a partially completed export. It takes no lock, creates no directories, and mutates nothing — you can run it while a real export is in progress. ## Concurrent runs One run per export root. A second run against the same `--out` exits 3 and names the holder's pid, host and start time. `flock` releases on process death, `kill -9` included, so there is no stale lock to clean up. Exporting two *different* groups at the same time works, but each needs its own `--session` as well as its own `--out`: the session file is a SQLite database and is locked too, because two Telethon clients sharing one session is a corruption hazard Telethon itself warns about. `--dry-run` takes no lock at all, so it can always be run against an export that is currently in progress. ## Exit codes | Code | Meaning | |---|---| | 0 | success | | 1 | unexpected error, or a bad argument | | 2 | `--dry-run` says it will not fit | | 3 | disk exhausted, or another run holds the lock | | 4 | session invalidated — re-login | | 5 | lost access to the group | | 6 | flood wait exceeded `--max-flood-wait` (only when you set it) | | 7 | filter or chat mismatch against the existing state file | | 130 | interrupted — re-run to resume | ## Development ```bash .venv/bin/pip install -r requirements.txt -e ".[dev]" .venv/bin/pytest ``` The whole suite runs offline against synthetic message stubs — no credentials, no network. Note that `messages.jsonl` contains message text and sender names from other people; the export tree is gitignored and the runtime check refuses an un-ignored path inside a repo, but check `git status` before any commit.