mirror of
https://github.com/tiennm99/telegram-exporter.git
synced 2026-10-11 03:13:49 +00:00
docs: document credentials, resume semantics and flood-wait behavior
Covers why the Bot API structurally cannot do this, what .env may and may not hold, why the export directory is a chat id, and the sidecar's union-over-records rule - reading it as last-write-wins reports a narrower file list than what is on disk after a filter change. Also states plainly that flood waits have no ceiling by default: the tool sleeps as long as Telegram demands and logs the computed wake time, so a long sleep reads as a sleep rather than a hang.
This commit is contained in:
tiennm99 committed
1 parent
5bf0992fe6
commit
30f68b43ff
1 file changed
+274
-1
@@ -1 +1,274 @@
|
||||
# telegram-exporter
|
||||
# telegram-exporter
|
||||
|
||||
`tg-export` downloads **all media from a Telegram group** to local disk, one
|
||||
folder per logical post (albums collapsed), with a `messages.jsonl` sidecar that
|
||||
links every file back to its message. Resumable, filterable, with a dry-run size
|
||||
estimate.
|
||||
|
||||
```bash
|
||||
export TG_API_ID=... TG_API_HASH=...
|
||||
|
||||
tg-export --group @mygroup --out ./exports --dry-run # what would it fetch?
|
||||
tg-export --group @mygroup --out ./exports # fetch it
|
||||
```
|
||||
|
||||
Kill it at any point and re-run the same command; it resumes without
|
||||
re-downloading and without gaps.
|
||||
|
||||
## Why the Bot API cannot do this
|
||||
|
||||
This is the first question most people ask, so: a bot cannot read a group's
|
||||
message history at all, and the Bot API caps file downloads at 20 MB. Both limits
|
||||
are structural, not configuration. Exporting history therefore requires MTProto
|
||||
with a **user account**, which is what Telethon provides.
|
||||
|
||||
## Install
|
||||
|
||||
Python 3.12+.
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
.venv/bin/pip install -r requirements.txt -e .
|
||||
.venv/bin/tg-export --help
|
||||
```
|
||||
|
||||
`requirements.txt` pins the exact Telethon version the API probes were verified
|
||||
against. Telethon 2.0 is alpha with a renamed API surface, hence the `<2` bound.
|
||||
|
||||
`cryptg` is **optional and not installed**. It is a native extension sitting in
|
||||
Telethon's AES path, so it sees the auth key and every plaintext byte. It ships an
|
||||
aarch64 wheel (no build step), and the reason to trust it *if* you adopt it is
|
||||
that the Telethon author maintains it — not that it is "CPU-side only". Do not add
|
||||
a supply-chain root for a throughput problem you have not measured.
|
||||
|
||||
## Credentials
|
||||
|
||||
Create an app at <https://my.telegram.org> → API development tools.
|
||||
|
||||
| Credential | Source | Why |
|
||||
|---|---|---|
|
||||
| `api_id`, `api_hash` | env `TG_API_ID` / `TG_API_HASH` only | needed every run; revocable |
|
||||
| phone | prompt; `TG_PHONE` optional | PII, not a secret |
|
||||
| login code | **prompt only** | single-use, 5-minute TTL |
|
||||
| 2FA password | **`getpass` only** — no env, no file, no flag | once per session lifetime |
|
||||
|
||||
`.env` **may** contain `TG_API_ID`, `TG_API_HASH`, `TG_PHONE`. It **must not**
|
||||
contain the 2FA password, a login code, or session data. The app does not read
|
||||
`.env` — your shell already does that:
|
||||
|
||||
```bash
|
||||
set -a; . ./.env; set +a
|
||||
```
|
||||
|
||||
### First run and the session file
|
||||
|
||||
The first run prompts for your phone, the login code Telegram sends you, and your
|
||||
2FA password if you have one. Later runs are non-interactive.
|
||||
|
||||
The session file defaults to
|
||||
`${XDG_STATE_HOME:-~/.local/state}/tg-export/default.session` — **outside the
|
||||
repo**, so the safe location needs no opt-in. It and every SQLite sibling are mode
|
||||
`0600`, enforced by a `umask(0o077)` set before anything can create a file, plus a
|
||||
post-login assertion. A `chmod` after connecting would be too late: SQLite writes
|
||||
the auth key *during* login.
|
||||
|
||||
If you point `--session` or `--out` inside a git repo at a path that is not
|
||||
gitignored, the run refuses with an explanation. Deleting a session file does
|
||||
**not** revoke it server-side — use Telegram → Settings → Devices for that.
|
||||
|
||||
## Output layout
|
||||
|
||||
```
|
||||
exports/g-1001234567890/ <- directory name is the chat id, never the group title
|
||||
title.txt <- the human-readable name lives here, as data
|
||||
1042/ <- one album (shared grouped_id); folder = first member id
|
||||
1042_photo.jpg <- filename keyed on message id, not a positional index
|
||||
1043_photo.jpg
|
||||
1044_video.mp4
|
||||
1047/ <- a standalone message
|
||||
1047_document.pdf
|
||||
messages.jsonl
|
||||
.export-state.json
|
||||
.export.lock
|
||||
```
|
||||
|
||||
The directory is `g<chat_id>` and not the group title because the title is
|
||||
server-supplied and editable by any admin. A title of `..` or `a/b` would be a
|
||||
path-traversal vector that no filename sanitizer could catch, since the escape
|
||||
happens in a parent component. Naming the directory after an int makes that
|
||||
structurally impossible — and also means a mid-export group rename cannot orphan
|
||||
your download. The cost is that `ls exports/` shows numbers.
|
||||
|
||||
Filenames are keyed on the **message id** rather than a position in the album,
|
||||
because a positional index shifts when a member message is deleted between runs,
|
||||
silently orphaning files already on disk.
|
||||
|
||||
## Resume semantics
|
||||
|
||||
`.export-state.json` holds one cursor, and it means:
|
||||
|
||||
> every post with `max_message_id <= cursor_id` has been handled **under these
|
||||
> filters**.
|
||||
|
||||
Consequences worth knowing:
|
||||
|
||||
- The cursor is written only after a post is completely handled, and never from an
|
||||
error path. It can never point past in-flight work.
|
||||
- Filters are stored verbatim. Changing them refuses the run and prints exactly
|
||||
what changed, because the cursor's meaning depends on them:
|
||||
|
||||
```
|
||||
filter mismatch:
|
||||
types: ['photo', 'video'] -> ['video']
|
||||
max_size: 100000000 -> (none)
|
||||
re-run with the original filters, or --reset-state to start over
|
||||
```
|
||||
|
||||
- `--reset-state` discards the cursor and rotates `messages.jsonl` to
|
||||
`messages-<utc-timestamp>.jsonl`. It does **not** delete downloaded files —
|
||||
dedupe reclaims them on the next run, and deleting 30 GB to change a filter
|
||||
would be hostile.
|
||||
- `--limit N` is *not* part of the filter set. It changes when we stop, not what a
|
||||
post contains, so a `--limit 20` test run is a genuine resumable partial export
|
||||
that does not poison later full runs.
|
||||
- `completed_at` is set only when a run exhausts the history cleanly. It is the
|
||||
only way to answer "did my 20-hour export actually finish?".
|
||||
- **A file that fails is recorded and skipped, and the cursor moves past it.** The
|
||||
run continues and the exit summary names the count (`N files failed (see
|
||||
messages.jsonl)`). Media that was deleted or expired is genuinely gone, but a
|
||||
file that exhausted its retries is not re-attempted by a later resume — grep
|
||||
`errors` in `messages.jsonl` to find them, and use `--reset-state` to re-drive
|
||||
that range if it matters.
|
||||
|
||||
## `messages.jsonl`
|
||||
|
||||
```json
|
||||
{"message_id":1043,"post_id":1042,"grouped_id":88,"date":"2026-07-04T12:00:00+00:00",
|
||||
"sender_id":777,"sender_name":"Alice","caption":"beach trip","reply_to":1001,
|
||||
"files":[{"path":"1042/1043_photo.jpg","size":1234567,"name":"photo.jpg",
|
||||
"mime":"image/jpeg","kind":"photo","declared_size":1234999}],
|
||||
"errors":[]}
|
||||
```
|
||||
|
||||
**Each record is an append-only event: "the files written for this message in this
|
||||
run."** So the correct way to read the sidecar is the **union of all records for a
|
||||
`message_id`**, never last-write-wins — after a filter change, the most recent
|
||||
record deliberately describes a narrower slice than what is on disk.
|
||||
|
||||
- `size` is bytes on disk; `declared_size` appears only when the server's number
|
||||
differs (see the photo note below).
|
||||
- Paths are relative to the export root.
|
||||
- `sender_name` is filled in only when the sender was already cached. Resolving it
|
||||
otherwise would mean one extra RPC per message — 100k RPCs and a flood ban on a
|
||||
large group, in exchange for a display string.
|
||||
- A partial final line from a host crash is truncated and warned about at startup.
|
||||
- With `--include-text`, text-only messages get the same record shape with
|
||||
`"files": []`. Distinguish media from context by `files` being empty, not by a
|
||||
record type.
|
||||
|
||||
## Flags
|
||||
|
||||
```
|
||||
tg-export --group <id|@username|link> --out DIR
|
||||
[--dry-run] [--since YYYY-MM-DD] [--until YYYY-MM-DD]
|
||||
[--types photo,video,document,audio,voice] [--max-size 100MB]
|
||||
[--include-text] [--limit N] [--session PATH] [--reset-state]
|
||||
[--min-free 2GiB] [--max-flood-wait SECONDS] [--verbose]
|
||||
```
|
||||
|
||||
- `--include-text` is off by default. When set, text-only messages are recorded in
|
||||
the sidecar with an empty `files[]` so captions and replies keep their
|
||||
surrounding conversation. They never create a folder and never download
|
||||
anything. It **is** part of the stored filter set — it changes what a post
|
||||
contains — so toggling it requires `--reset-state`.
|
||||
- `--verbose` raises the log level for *this tool only*. The `telethon` logger is
|
||||
pinned at `WARNING` unconditionally, because its request logging carries
|
||||
`api_id`, `phone_number` and `phone_code_hash`. There is deliberately no
|
||||
`--debug-telethon`.
|
||||
- A link preview is not media: a message containing a URL is not treated as an
|
||||
attachment, so preview thumbnails are never downloaded.
|
||||
|
||||
## Flood waits have no ceiling by default
|
||||
|
||||
Telegram throttles by telling the client to wait, and those waits escalate from
|
||||
seconds to hours if you keep pushing. By default `tg-export` **sleeps as long as
|
||||
Telegram demands**, so an unattended export completes rather than failing fast.
|
||||
|
||||
Every wait is logged at `WARNING` with its duration **and its computed wake
|
||||
time**:
|
||||
|
||||
```
|
||||
2026-08-11 07:42:11 WARNING flood wait 3600s - sleeping until 2026-08-11T08:42:11+00:00
|
||||
```
|
||||
|
||||
A long sleep is a sleep, not a hang. Read the log before assuming otherwise. Pass
|
||||
`--max-flood-wait SECONDS` to exit cleanly and resumably (code 6) instead.
|
||||
|
||||
## Why downloads are sequential
|
||||
|
||||
Telethon does not parallelize a single transfer, and running several at once
|
||||
mainly accelerates flood-wait escalation — a 7-second wait becomes a 4-hour
|
||||
lockout. There is deliberately no `--workers` flag, and an AST test in the suite
|
||||
fails if `gather`, `create_task`, `to_thread` or a thread pool ever appears in
|
||||
`src/`.
|
||||
|
||||
## Disk space
|
||||
|
||||
`--dry-run` reports remaining posts, files and bytes by media type against free
|
||||
space, and exits 2 if it will not fit.
|
||||
|
||||
**That verdict is advisory.** The enforcement is a per-file check before each
|
||||
download, comparing free space against *this file's* size plus the `--min-free`
|
||||
reserve. So a `SHORT` verdict does not mean the run is unsafe — it means the run
|
||||
will stop partway with a clean, resumable cursor.
|
||||
|
||||
Photo totals are labelled approximate because a photo's declared `.size`
|
||||
describes the size variant Telethon selected. Documents, video and audio carry an
|
||||
exact size. Media reporting no size at all is downloaded anyway and counted on its
|
||||
own `unknown size` line rather than as zero, so the estimate never hides an
|
||||
unbounded error.
|
||||
|
||||
`--dry-run` reads the cursor and measures from where a real run would start, so
|
||||
`tg-export --dry-run && tg-export` works on a partially completed export. It takes
|
||||
no lock, creates no directories, and mutates nothing — you can run it while a real
|
||||
export is in progress.
|
||||
|
||||
## Concurrent runs
|
||||
|
||||
One run per export root. A second run against the same `--out` exits 3 and names
|
||||
the holder's pid, host and start time. `flock` releases on process death, `kill -9`
|
||||
included, so there is no stale lock to clean up.
|
||||
|
||||
Exporting two *different* groups at the same time works, but each needs its own
|
||||
`--session` as well as its own `--out`: the session file is a SQLite database and
|
||||
is locked too, because two Telethon clients sharing one session is a corruption
|
||||
hazard Telethon itself warns about.
|
||||
|
||||
`--dry-run` takes no lock at all, so it can always be run against an export that
|
||||
is currently in progress.
|
||||
|
||||
## Exit codes
|
||||
|
||||
| Code | Meaning |
|
||||
|---|---|
|
||||
| 0 | success |
|
||||
| 1 | unexpected error, or a bad argument |
|
||||
| 2 | `--dry-run` says it will not fit |
|
||||
| 3 | disk exhausted, or another run holds the lock |
|
||||
| 4 | session invalidated — re-login |
|
||||
| 5 | lost access to the group |
|
||||
| 6 | flood wait exceeded `--max-flood-wait` (only when you set it) |
|
||||
| 7 | filter or chat mismatch against the existing state file |
|
||||
| 130 | interrupted — re-run to resume |
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
.venv/bin/pip install -r requirements.txt -e ".[dev]"
|
||||
.venv/bin/pytest
|
||||
```
|
||||
|
||||
The whole suite runs offline against synthetic message stubs — no credentials, no
|
||||
network. Note that `messages.jsonl` contains message text and sender names from
|
||||
other people; the export tree is gitignored and the runtime check refuses an
|
||||
un-ignored path inside a repo, but check `git status` before any commit.
|
||||
Reference in new issue
Block a user