diff --git a/plans/260811-1411-telegram-group-media-exporter/phase-01-verification-spike.md b/plans/260811-1411-telegram-group-media-exporter/phase-01-verification-spike.md new file mode 100644 index 0000000..7904a92 --- /dev/null +++ b/plans/260811-1411-telegram-group-media-exporter/phase-01-verification-spike.md @@ -0,0 +1,139 @@ +--- +phase: 1 +title: "Verification Spike" +status: partial +priority: P1 +dependencies: [] +effort: "~45 min" +--- + +# Phase 1: Verification Spike + +## Overview + +Throwaway script settling the unverified items in the research confidence ledger against the live API, before production code depends on them. The docs page truncated mid-signature; guessing means silent resume bugs discovered after a 40 GB run. + +**Ordering is load-bearing in this phase.** `.gitignore` is committed *first*, before anything can create a credential. The spike uses its own session path so phase 2's "first production login" remains a real, verifiable event. + +## Requirements + +- Functional: answer all six probes; record findings; amend contradicted phases +- Non-functional: throwaway code, not shipped; no credential exists before `.gitignore` is committed; one test download only + +## Probes + +### P1 — `offset_id` direction under `reverse=True` (highest risk) + +Semantics invert between sweep directions. Wrong here means resume silently skips or re-downloads everything. + +```python +ids = [m.id async for m in client.iter_messages(entity, limit=5)] # newest-first +mid = sorted(ids)[2] +fwd = [m.id async for m in client.iter_messages(entity, reverse=True, offset_id=mid, limit=3)] +assert all(i > mid for i in fwd), f"offset_id not exclusive-lower-bound under reverse: {fwd} vs {mid}" +``` + +Record whether `mid` itself is included. **Probe `min_id=mid` the same way** — it is documented as an exclusive lower bound and may be the more robust parameter regardless. The result decides whether the cursor stores "last handled id" or "next id to fetch". + +### P2 — 2FA / login flow + +Record whether `SessionPasswordNeededError` is raised and whether `client.start(phone=..., password=)` handles it. Confirm a second run against the same session needs no interaction. + +2FA password is `getpass`-only — never argv, never env, never a file. (This supersedes any earlier wording permitting env.) + +### P3 — `message.file` attribute surface + +**Open question, not a confirmation.** For one photo, one video, one document, one voice note, one text-only message, and anything unusual in reach (poll, webpage preview, expiring media): + +```python +f = msg.file +print(msg.id, msg.grouped_id, type(msg.media).__name__, + getattr(f, 'name', None), getattr(f, 'ext', None), + getattr(f, 'size', None), getattr(f, 'mime_type', None)) +``` + +Answer: **for which media kinds is `.size` absent?** Do not assume it is always present — phases 3, 5, and 6 each need a rule for `None`. + +### P4 — `download_media` return value and file-object support + +- Download one photo and one document with `file=`; record the exact return type/value. Phase 5 uses this as the source of truth for `os.replace`. +- Test whether `download_media(msg, file=)` works across media branches. Phase 5 needs to own the fd to `fsync` it before rename. If it does not work, the fallback is download-to-path then re-open and fsync. + +### P5 — photo `.size` vs bytes on disk (NEW — ledger item the original probe set missed) + +```python +p = await client.download_media(photo_msg, file=str(tmp)) +print("declared", photo_msg.file.size, "actual", os.path.getsize(p), + "delta", os.path.getsize(p) - photo_msg.file.size) +``` + +Phase 6 documents photo sizes as approximate while phase 5 originally used exact equality as a hard gate — that contradiction was the poison pill this probe closes. The resolved design is answer-independent; the answer only improves report labelling and validator tuning. + +### P6 — album adjacency across a pagination boundary + +The original probe capped `limit ≤ 5`, which cannot cross Telethon's ~100-message page boundary and so could not test the thing it existed to test. **Cap downloads at 1; do not cap history reads.** + +```python +rows = [(m.id, m.grouped_id) async for m in + client.iter_messages(entity, reverse=True, offset_id=start, limit=350)] +assert len(rows) >= 250, "window too small to cross a page boundary — probe is vacuous" +ids = [i for i, _ in rows] +assert ids == sorted(ids), "sweep is not strictly ascending" +runs = [k for k, _ in itertools.groupby(rows, key=lambda r: r[1])] +gids = [g for g in runs if g is not None] +assert len(gids) == len(set(gids)), f"a grouped_id occupies >1 run — albums DO split: {gids}" +``` + +Coverage is probabilistic (3+ pages, several albums). The deterministic guarantee is phase 3's runtime tripwire; this probe is the cheap prior. + +## Related Code Files + +- Create: `.gitignore` (final version, committed first), `requirements.txt` +- Create: `spike/probe_telethon_api.py` (throwaway, gitignored) +- Create: `plans/reports/spike-260811-1411-telethon-api-verification-report.md` + +## Implementation Steps + +1. Write the **final** `.gitignore` and commit it. Nothing else exists yet: + ``` + .venv/ + *.session + *.session-journal + *.session-wal + *.session-shm + spike/ + exports/ + .export-state.json + .export.lock + __pycache__/ + *.egg-info/ + .env + ``` +2. **Gate — must pass before step 3:** `git check-ignore -q .venv spike/ && echo OK` (exit 0) +3. `python3 -m venv .venv && .venv/bin/pip install "telethon>=1.41,<2"`, then `.venv/bin/pip freeze > requirements.txt` — this records the exact version the probes verified +4. `umask 077; mkdir spike` +5. Obtain `api_id`/`api_hash` from my.telegram.org; export as `TG_API_ID`/`TG_API_HASH` (never hardcode) +6. Write and run `spike/probe_telethon_api.py` (P1–P6), session at `spike/probe.session` +7. Write the findings report, marking each ledger item CONFIRMED or CORRECTED, and recording `telethon.__version__` +8. **If any finding contradicts the plan, amend the affected phase file before proceeding** + +## Success Criteria + +- [ ] `.gitignore` is committed before any venv, session, or credential exists — verified by `git log` +- [ ] `git check-ignore -q spike/ .venv` exits 0 before step 3 runs +- [ ] P1: `offset_id` inclusivity/direction under `reverse=True` recorded, and `min_id` compared +- [ ] P2: 2FA flow documented; second run confirmed non-interactive +- [ ] P3: `.size` availability tabulated per media kind, including any kind where it is absent +- [ ] P4: `download_media` return value recorded; file-object support determined +- [ ] P5: photo declared-vs-actual size delta recorded +- [ ] P6: ≥250 messages swept, ascending order asserted, no `grouped_id` occupying two runs +- [ ] `requirements.txt` records the verified Telethon version +- [ ] Findings report written; contradicted phases amended + +## Risk Assessment + +- **Wrong `offset_id` assumption reaches production** → the point of this phase; assertions must run, not be eyeballed. +- **Credential created before `.gitignore`** → step 1 precedes step 3, with an explicit gate between them. +- **Probe capped too small to test what it claims** → P6 requires ≥250 messages and asserts it. History reads are cheap (~4 requests); only *downloads* are meaningfully rate-limited. +- **Spike session left live** → deleting `spike/probe.session` does **not** revoke server-side authorization. Phase 7 cleanup must terminate it in Telegram → Settings → Devices. +- **Credentials leaked into the spike script** → env vars only; `spike/` gitignored from commit one. diff --git a/plans/260811-1411-telegram-group-media-exporter/phase-02-skeleton-and-session-auth.md b/plans/260811-1411-telegram-group-media-exporter/phase-02-skeleton-and-session-auth.md new file mode 100644 index 0000000..50f97a2 --- /dev/null +++ b/plans/260811-1411-telegram-group-media-exporter/phase-02-skeleton-and-session-auth.md @@ -0,0 +1,215 @@ +--- +phase: 2 +title: "Skeleton and Session Auth" +status: complete +priority: P1 +dependencies: [1] +effort: "~2h" +--- + +# Phase 2: Skeleton and Session Auth + +## Overview + +Package skeleton, the authenticated Telethon client, the flood-wait primitives every network call routes through, and the process-level safety rails: umask, logging policy, git-safety assertion, and the single-instance lock. + +## Requirements + +- Functional: one-time interactive login; later runs non-interactive; `tg-export --help` after `pip install -e .` +- Non-functional: no file is ever group/world-readable at any instant; no credential in argv, env-where-forbidden, or logs; two concurrent runs cannot corrupt each other + +## Architecture + +### Process rails — `cli.main()`, before anything else + +```python +os.umask(0o077) # FIRST statement. Covers .session and every sqlite sibling, + # export tree, messages.jsonl, .part, state, lock — one line, no per-file chmod. +``` + +Verified empirically: `sqlite3.connect()` on a fresh path yields `0644` under a default umask, and the auth key is written *during* login — so a post-hoc `chmod` is provably too late. With `umask(0o077)` set first, the db and its `-journal` are both `0600`. + +### Session path + +```python +SUFFIX = ".session" + +def prepare_session_path(p: Path) -> Path: + # Telethon appends '.session' when absent — chmod-ing the un-suffixed path protects nothing + p = p if p.name.endswith(SUFFIX) else p.with_name(p.name + SUFFIX) + p.parent.mkdir(parents=True, exist_ok=True) + try: + os.close(os.open(p, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)) + except FileExistsError: + os.chmod(p, 0o600) + return p + +def assert_session_private(p: Path) -> None: # this is the test, not a checkbox + for f in p.parent.glob(p.name + "*"): + if stat.S_IMODE(f.stat().st_mode) & 0o077: + raise SystemExit(f"insecure mode on {f}: {oct(f.stat().st_mode)}") +``` + +Default path `${XDG_STATE_HOME:-~/.local/state}/tg-export/.session` — **outside the repo**, so the safe location is the default and the git check only bites deliberate choices. + +### Git safety — robust to `--out` / `--session` moving the targets + +Adding `.gitignore` patterns cannot help when the operator can point a flag anywhere. Assert at runtime: + +```python +def assert_not_committable(p: Path) -> None: + """No-op outside a git work tree. rc 0=ignored, 1=NOT ignored, 128=not a repo.""" + try: + rc = subprocess.run(["git", "check-ignore", "-q", str(p)], + cwd=p.parent, capture_output=True).returncode + except FileNotFoundError: + return + if rc == 1: + raise SystemExit(f"refusing: {p} is inside a git repo and is not gitignored.\n" + f"add it to .gitignore, or choose a path outside the repo.") +``` + +Called on the session path and export root at startup. Exit codes verified empirically, including on nonexistent paths. + +### Single-instance lock + +```python +@contextmanager +def exclusive(path: Path): + """flock releases on process death, kill -9 included — no stale-lock cleanup to get wrong.""" + path.parent.mkdir(parents=True, exist_ok=True) + f = open(path, "a+") + try: + fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError: + f.seek(0) + raise SystemExit(f"busy: another tg-export holds {path} [{f.read().strip() or '?'}]") + f.seek(0); f.truncate() + f.write(f"pid={os.getpid()} host={socket.gethostname()} started={now_iso()}\n"); f.flush() + try: + yield + finally: + f.close() +``` + +Two call sites: `/.export.lock` (covers `.part` files, state, sidecar) and `.lock` (two Telethon clients on one SQLite session is what Telethon itself warns about). Per-root, not global — exporting two different groups concurrently is legitimate. + +Contention: fail fast, exit 3, no wait, no `--force`. The message names pid/host/start-time, so "did it hang?" gets an answer instead of a second process. **Dry-run takes no lock** — it writes nothing. + +### Logging policy + +```python +def configure_logging(verbose: bool) -> None: + logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s") + logging.getLogger("telegram_exporter").setLevel(logging.DEBUG if verbose else logging.INFO) + logging.getLogger("telethon").setLevel(logging.WARNING) # unconditional floor + logging.getLogger("asyncio").setLevel(logging.WARNING) +``` + +Never touch the root level. `--verbose` means *our* logger only — telethon's request logging carries `api_id`, `phone_number`, and `phone_code_hash`. No `--debug-telethon` flag. + +Note: CPython does not print local variables in tracebacks, so no exception-scrubbing machinery is needed. Just don't install `rich` or `better-exceptions`. + +### Credential policy — one table, referenced by every other doc + +| Credential | Source | Why | +|---|---|---| +| `api_id`, `api_hash` | env `TG_API_ID` / `TG_API_HASH` only | needed every run; revocable at my.telegram.org | +| phone | prompt; `TG_PHONE` optional | PII, not a secret | +| login code | **prompt only** | single-use, 5-minute TTL | +| 2FA password | **`getpass` only** — no env, no file, no flag | once per session lifetime; prompting costs nothing | + +`.env` **may** contain `TG_API_ID`, `TG_API_HASH`, `TG_PHONE`. It **must not** contain the 2FA password, login code, or session data. The app does **not** read `.env` — no `python-dotenv` to read a file the shell already handles (`set -a; . ./.env; set +a`). + +### Flood-wait primitives — two, not one + +The single-value wrapper cannot protect the sweep: `iter_messages` is consumed with `async for`, and Telethon raises `FloodWaitError` on the *next page fetch*, deep inside the loop — so it escapes a wrapper that only guards construction, and re-invoking the factory would restart the sweep from `after_id`. + +```python +def _flood_sleep(e, max_wait_s, where=""): + """max_wait_s is None by default -> sleep however long Telegram demands. + Always log the WAKE TIME, not just the duration: an unattended 4-hour sleep must + read as a sleep, not as a hang. This is the mitigation for the no-ceiling default.""" + if max_wait_s is not None and e.seconds > max_wait_s: + raise Abort(f"flood wait {e.seconds}s exceeds --max-flood-wait {max_wait_s}s", 6) + wake = datetime.now(UTC) + timedelta(seconds=e.seconds) + log.warning("flood wait %ss%s — sleeping until %s", e.seconds, where, wake.isoformat()) + return asyncio.sleep(e.seconds + random.uniform(1.0, 3.0)) + +async def with_flood_retry(make_coro, *, max_wait_s: int | None = None): + """Download path. Never tight-retry — retries during an active wait are fresh violations.""" + while True: + try: + return await make_coro() + except FloodWaitError as e: + await _flood_sleep(e, max_wait_s) + +async def aiter_with_flood_retry(make_agen, *, start_id, max_wait_s: int | None = None): + """Sweep path. Catches mid-iteration waits and rebuilds the generator from the + last id actually yielded, so no message is skipped or replayed.""" + last = start_id + while True: + try: + async for item in make_agen(last): + last = item.id + yield item + return + except FloodWaitError as e: + await _flood_sleep(e, max_wait_s, where=f" mid-sweep after id {last}") +``` + +`max_wait_s=None` is the default per the validation decision: unattended completion is preferred over failing fast. The wake-time log is what keeps that choice diagnosable. + +Also set `client.flood_sleep_threshold = 120` so routine short waits are absorbed inside Telethon. + +## Related Code Files + +- Create: `pyproject.toml`, `README.md` +- Create: `src/telegram_exporter/{__init__,cli,session}.py` +- Create: `tests/{__init__,test_logging,test_lock}.py` +- Modify: `.gitignore` (verify coverage; phase 1 created it) + +## Implementation Steps + +1. `pyproject.toml` — `telethon>=1.41,<2`, `[project.scripts] tg-export = "telegram_exporter.cli:main"`, hatchling backend, `dev = ["pytest>=8"]`. Install with `pip install -r requirements.txt -e .` so the phase-1-verified version is what's used. +2. **Verify** (do not recreate) that `.gitignore` covers the production session path and export root. +3. `cli.py`: `os.umask(0o077)` as the first statement, then `configure_logging`, argparse, `assert_not_committable`, locks, dispatch. +4. `session.py`: credential loading with an actionable error pointing at my.telegram.org; `prepare_session_path`; client construction; login (phone/code prompts, 2FA via `getpass`); `assert_session_private` after connect; async context manager guaranteeing `disconnect()` — SQLite state is not flushed without it. +5. `session.py`: both flood primitives and the `Abort` exception type. +6. Entity resolution for `--group`: numeric id, `@username`, `t.me` link. Use `telethon.utils.get_peer_id(entity)` as the canonical int for the export root. Clear message when the account is not a member. +7. Full flag surface (later phases wire their own): + ``` + tg-export --group --out DIR + [--dry-run] [--since YYYY-MM-DD] [--until YYYY-MM-DD] + [--types photo,video,document,audio,voice] [--max-size 100MB] + [--include-text] [--limit N] [--session PATH] [--reset-state] + [--min-free 2GiB] [--max-flood-wait SECONDS] [--verbose] + ``` + `--max-flood-wait` is **unset by default — the tool sleeps through any flood wait, however long** (validation decision). Passing a number imposes a ceiling above which the run exits cleanly with a resumable cursor (exit 6). + + + +8. Tests: `test_logging.py` (telethon logger stays ≥ WARNING even with `verbose=True`), `test_lock.py` (subprocess holds the lock; second acquire exits 3). + +## Success Criteria + +- [ ] `pip install -r requirements.txt -e ".[dev]"` succeeds on Python 3.12 / aarch64 +- [ ] `tg-export --help` prints the full flag surface +- [ ] Production session path does not exist before first login (asserted) — phase 1's spike session is separate +- [ ] After login: session and every sibling matching `session*` are mode `0600` +- [ ] Second run is non-interactive: `tg-export --group G --limit 0 prev, f"sweep not strictly ascending: {msg.id} after {prev}" + prev = msg.id + if buf and (msg.grouped_id is None or msg.grouped_id != buf[0].grouped_id): + yield _close(buf, closed, filters); buf = [] + if msg.grouped_id is None: + yield _close([msg], closed, filters) + else: + buf.append(msg) + if buf: + yield _close(buf, closed, filters) + +def _close(buf, closed, filters) -> Post: + gid = buf[0].grouped_id + if gid is not None: + if gid in closed: # album-split tripwire + raise AlbumSplitError( + f"grouped_id {gid} reopened at msg {buf[0].id} after being closed — " + f"album split across a non-adjacent boundary. Do not trust this export.") + closed.add(gid) + return Post(post_id=buf[0].id, + messages=[m for m in buf if _keep(m, filters)], + max_message_id=buf[-1].id) +``` + +**Empty posts are yielded, not skipped.** This is the resolved answer to a decision the plan previously deferred. Skipping them stalls the cursor: a `--types video` run over a photo-heavy tail would advance `cursor_id` only at videos, so every resume re-sweeps tens of thousands of messages to download nothing. The "phantom post count" objection is a *counting* bug, fixed with one line in each consumer (`if not post.messages: continue` before counting), not a design objection. + +**Album-split tripwire.** Consecutiveness of `grouped_id` members in a sweep is assumed by the whole design and was never verified by any source. The tripwire converts a silent split — two folders for one album, no exception, no size mismatch, dedupe reporting complete — into a loud failure. Memory is one int per album (~6 MB at 100k albums); the "flat memory" criterion is restated honestly as O(albums) rather than pretending it is O(1). + +Tracking only `last_flushed_gid` was rejected: it catches `A B A` but misses `A B C A`, saving 6 MB and losing most of the coverage. + +### `media_size` — one rule, three consumers (DRY) + +```python +def media_size(msg) -> int | None: + """Single source of truth. Phase 1 P3 determines which kinds return None.""" +``` + +| Consumer | Rule for `None` | +|---|---| +| `--max-size` filter | `size is None or size <= limit` → **include**, WARN once. Silent drops violate the no-silent-gaps ethos; unknown-size media is pathological and near-always small. | +| download validator | accept (nothing to check) | +| estimator | **not** `or 0` — counted in a separate `unknown size: N files (not counted)` line. An unbounded estimate error must be visible. | +| disk guard | treated as 0, absorbed by the reserve | + +Comparing `None <= limit` unguarded raises `TypeError` inside the generator and kills a 20-hour export — that is what this helper exists to prevent. + +### Filters + +| Flag | Predicate | +|---|---| +| `--types` | media kind ∈ set; kind from `msg.media` type + document attributes | +| `--since` / `--until` | `msg.date` in range, UTC-aware on both sides | +| `--max-size` | via `media_size` per the table above | +| `--include-text` | when set, keep text-only messages too (sidecar context); default off | + + + +**`--include-text`** (validation decision). Default off: messages with no media are dropped. When set, text-only messages survive `_keep` and reach the sidecar with an empty `files[]`, so captions and replies have surrounding conversation. They never create a post folder and never reach `download_one`. + +It **is** part of the stored filter set — it changes what a post contains, so running without it and then with it against the same cursor would silently skip every text record. + +`--limit N` counts **posts with files** (counting all posts would let it terminate instantly inside a filtered region, and would now also be skewed by text-only posts) and is *not* a filter — it changes when we stop, not what a post contains, so it stays out of the state file's filter set. + +## Related Code Files + +- Create: `src/telegram_exporter/traversal.py` +- Create: `tests/test_traversal.py` +- Modify: `src/telegram_exporter/cli.py` (wire `--types`, `--since`, `--until`, `--max-size`, `--limit`) + +## Implementation Steps + +1. Define `Post`, `Filters`, `AlbumSplitError`; parse CLI values into `Filters` (size-string and ISO-date parsers) +2. Implement media-kind classification and `media_size` +3. Implement `iter_posts` per the sketch, honoring phase 1 P1's confirmed `offset_id`/`min_id` semantics +4. Consume the sweep through `aiter_with_flood_retry` — not the single-value wrapper +5. Unit test with **synthetic message stubs** (no network): objects with `id`, `grouped_id`, `date`, `file` +6. Test the tripwire explicitly: a synthetic stream where a `grouped_id` reappears must raise + +## Success Criteria + +- [ ] A 3-photo album yields exactly one `Post` with 3 messages +- [ ] Two adjacent albums yield two distinct `Post`s, not one merged +- [ ] A standalone message between two albums yields its own `Post` +- [ ] An album at the very end of history is flushed +- [ ] `post_id` and `max_message_id` are unchanged when a filter removes some members — asserted +- [ ] A fully-filtered group still yields a `Post` (with `messages == []`) so the cursor advances +- [ ] Text-only messages contribute no files, and are dropped entirely unless `--include-text` is set +- [ ] With `--include-text`, a text-only message survives into `Post.messages` but creates no folder and no download +- [ ] `--limit N` counts posts with files, unaffected by `--include-text` +- [ ] A synthetic stream reopening a closed `grouped_id` raises `AlbumSplitError` +- [ ] A non-ascending synthetic stream trips the assertion +- [ ] `media_size` returning `None` does not raise in any of the three consumers +- [ ] Filters compose: `--types video --since 2026-01-01 --max-size 50MB` +- [ ] `--limit 5` stops after 5 posts *with files* +- [ ] Album buffer never exceeds 20 entries (asserted in code) + +## Risk Assessment + +- **Identity computed post-filter** → duplicate folders and re-downloads across filter sets. Invariant 1, plus phase 4's message-id filenames; both explicitly tested. +- **Album split across a pagination boundary** → silent two-folder corruption. Phase 1 P6 is the prior; the tripwire is the guarantee. Phase 3's earlier dismissal of this risk was an unsourced assertion guarded by a test that structurally could not fail. +- **Cursor stalls across filtered regions** → empty posts are yielded. Decided here, not deferred to implementation. +- **`None` size crashes the sweep** → `media_size` plus the stated rule per consumer. +- **Flood wait mid-sweep** → `aiter_with_flood_retry` resumes from the last yielded id. +- **Timezone-naive comparison** → `msg.date` is UTC-aware; parse `--since`/`--until` as UTC-aware or the comparison raises. diff --git a/plans/260811-1411-telegram-group-media-exporter/phase-04-naming-and-path-safety.md b/plans/260811-1411-telegram-group-media-exporter/phase-04-naming-and-path-safety.md new file mode 100644 index 0000000..a487fcb --- /dev/null +++ b/plans/260811-1411-telegram-group-media-exporter/phase-04-naming-and-path-safety.md @@ -0,0 +1,145 @@ +--- +phase: 4 +title: "Naming and Path Safety" +status: complete +priority: P1 +dependencies: [2] +effort: "~1h" +--- + +# Phase 4: Naming and Path Safety + +## Overview + +`paths.py` — **all** path construction, from the export root down to the filename leaf. Pure functions, no network, fully unit-testable, which is why it is its own module rather than a helper inside `downloader.py`. + +Renamed from `naming.py`: the module owns directory construction too. That rename is what turns the group-title fix from a patch into a structural property. + +## Requirements + +- Functional: derive the export root, post dir, and filename; sanitize; guarantee uniqueness within a post folder +- Non-functional: pure; deterministic across runs *and across filter sets* (dedupe depends on it) + +## The governing invariant + +> **Exactly one untrusted string ever becomes a path component: the filename leaf.** + +Every other component is an `int`, enforced by a type check. This makes `safe_join`'s "post_dir is trusted" precondition true *by type* rather than by assumption. + +Two untrusted strings were previously in play: + +1. `DocumentAttributeFilename` — attacker-controlled, supplied by arbitrary group members. `../../../.ssh/authorized_keys` is a cheap, real attack. Always was in scope. +2. **The group title** — server-supplied and editable by any group admin, used to build `exports//`. This was *not* in scope, and `safe_join` could never have caught it because the escape happens in the parent component, before the join. + +## Architecture + +```python +def export_root(out_dir: Path, peer_id: int) -> Path: + """peer_id from telethon.utils.get_peer_id(entity). int -> traversal impossible.""" + if not isinstance(peer_id, int): + raise TypeError(peer_id) + return (out_dir / f"g{peer_id}").resolve() # g-1001234567890 + +def post_dir(root: Path, post_id: int) -> Path: + if not isinstance(post_id, int): + raise TypeError(post_id) + return root / str(post_id) + +def derive_filename(msg) -> str: # note: no index parameter + return f"{msg.id}_{sanitize(stem_for(msg), fallback=kind(msg))}{ext_for(msg)}" + +def sanitize(name: str, *, fallback: str, max_bytes: int = 200) -> str: + """Reduce an untrusted string to a safe single path component.""" + +def safe_join(post_dir: Path, filename: str) -> Path: + """Join and assert containment. Raises on escape — never silently repairs.""" +``` + +### Why the directory is the chat id, not the title + +The `g` prefix avoids a leading `-` (a CLI foot-gun, not a security issue). The human-readable name is written once to `/title.txt` and printed in the run header — stored as data, never as a path. + +Rejected alternatives: a sanitized title slug needs a second sanitizer with different rules for directories, a fallback for the empty case, and a collision story — more code and more risk. A `-` hybrid is readable but strands the state file and the downloaded tree in the old directory when the group is renamed mid-export. + +One move fixes traversal, empty-slug collapse into the parent, cross-group collision, *and* rename-orphaning — and it is a net code deletion. The cost is that `ls exports/` shows numbers. + +### Why filenames key on `message_id`, not a positional index + +One Telegram message carries at most one media, so `message_id` cannot collide inside a post folder. Uniqueness becomes structural rather than positional. + +A positional index — even one computed pre-filter — is unstable under *history mutation*: if a member message is deleted between runs the group shrinks and every later index shifts, silently orphaning already-downloaded files. It would also force `Post` to carry the full pre-filter member list purely for naming, reintroducing the coupling Invariant 1 exists to remove. + +Accepted cost: `1042/1042_photo.jpg` is redundant for standalone posts. Worth it — files stay self-identifying when moved out of their folder, and `grep 1043 messages.jsonl` joins sidecar to disk with no index arithmetic. + +### Sanitization rules + +Derivation order: `msg.file.name` when present → `msg.file.ext`/mime with a kind-based stem (`photo`, `video`, `voice` carry no filename) → `.bin`. + +- Take `Path(name).name` only — discards every directory component, including `..` +- Strip NUL and control chars (`< 0x20`); replace path separators and reserved chars +- Strip leading dots and dashes (hidden files, argument-looking names) +- Collapse whitespace runs +- Truncate to `max_bytes` counted in **UTF-8 bytes, not characters** — ext4's limit is 255 bytes and unicode names overflow well before 255 chars. Preserve the extension. +- Empty or fully-stripped result → `fallback` + +Final defense in `safe_join`: + +```python +resolved = (post_dir / filename).resolve() +if not resolved.is_relative_to(post_dir.resolve()): + raise ValueError(f"path escape attempt: {filename!r}") +``` + +Sanitization should make this unreachable; reaching it means sanitization has a hole — hence raise, never repair. + +## Related Code Files + +- Create: `src/telegram_exporter/paths.py` +- Create: `tests/test_paths.py` + +## Implementation Steps + +1. Implement `sanitize` per the rules above +2. Implement `derive_filename` using phase 1 P3's findings on `message.file` availability +3. Implement `export_root` / `post_dir` with `isinstance` guards +4. Implement `safe_join` with the containment assertion +5. Write the table-driven tests below +6. `pytest tests/test_paths.py` must pass before phase 5 consumes this module + +## Test Table + +| Input | Expected | +|---|---| +| `../../../etc/passwd` | `passwd` | +| `a/b/c.jpg` | `c.jpg` | +| `..` / `.` / `""` | fallback | +| `\x00evil.jpg` | `evil.jpg` | +| `....jpg` | leading dots stripped | +| `-rf.jpg` | leading dash stripped | +| 300-char unicode stem | ≤200 UTF-8 bytes, extension intact | +| same msg, two different filter sets | **identical** filename | +| two media in one post | distinct names via distinct message ids | +| `safe_join(dir, "../x")` | raises `ValueError` | +| `export_root(out, "..")` | raises `TypeError` (not an int) | +| `export_root(out, "../../etc")` | raises `TypeError` | +| `post_dir(root, "1042; rm -rf")` | raises `TypeError` | + +## Success Criteria + +- [ ] Every row of the test table passes +- [ ] No sanitized name, joined to a post dir, resolves outside it +- [ ] `export_root` and `post_dir` reject every non-`int` input +- [ ] A group titled `..`, `a/b`, or `""` cannot influence the output path — there is no code path from title to path +- [ ] Truncation is byte-counted; a 300-char unicode name yields a filename ext4 accepts, extension intact +- [ ] `derive_filename` is deterministic across runs **and across filter sets** +- [ ] Tests run offline with no Telethon client + +## Risk Assessment + +- **Path traversal via filename** → sanitize + `safe_join` assertion + table tests. +- **Path traversal via group title** → structurally impossible: no code path exists from title to path component. +- **Filter-dependent filenames** → keyed on `message_id`; asserted by the two-filter-sets test row. +- **Index instability under message deletion** → no positional index exists. +- **Char-based truncation on unicode** → `OSError: File name too long` mid-export. Byte-counted. +- **Non-deterministic fallbacks** (e.g. a timestamp) → dedupe breaks and every re-run re-downloads. Fallbacks derive only from message id and kind. +- **Windows reserved names** (`CON`, `PRN`) → not applicable on Linux; deliberately skipped. diff --git a/plans/260811-1411-telegram-group-media-exporter/phase-05-downloader-state-and-resume.md b/plans/260811-1411-telegram-group-media-exporter/phase-05-downloader-state-and-resume.md new file mode 100644 index 0000000..f90fe3b --- /dev/null +++ b/plans/260811-1411-telegram-group-media-exporter/phase-05-downloader-state-and-resume.md @@ -0,0 +1,268 @@ +--- +phase: 5 +title: "Downloader State and Resume" +status: complete +priority: P1 +dependencies: [3, 4] +effort: "~4h" +--- + +# Phase 5: Downloader State and Resume + +## Overview + +The download path: fetch media sequentially, write durably, dedupe, persist the resume cursor and the `messages.jsonl` sidecar. This is where correctness under interruption is won or lost. + +Governing principle: **the filesystem is the state; the state file is a hint; the sidecar is the log.** Everything below follows from it. + +## Requirements + +- Functional: download each `Post`'s media into its folder; skip completed files; resume after any interruption; record every file and every failure +- Non-functional: concurrency 1; no partial file ever presented as complete; the cursor never points past an incomplete post + +## Architecture + +### Durability and dedupe are one decision, not two + +Telethon has no byte-level resume — `download_media` writes from offset 0 every call. So resumability lives at *message* granularity, and completeness is proven by **arrival at the target path**: + +``` +per file: write -> flush -> fsync(fd) -> os.replace(tmp, target) +per post: fsync(post_dir) -> sidecar.append + fsync -> state.commit (tmp -> fsync -> replace -> fsync(root)) +``` + +With `fsync` before `os.replace`, a file at the target path can only have arrived fully downloaded. Therefore: + +```python +if target.exists(): + if target.stat().st_size > 0: + return SKIPPED + target.unlink() # zero-byte junk from ENOSPC; re-fetch +``` + +**Existence, not size equality.** Size equality would only add detection of *external* corruption — a fabricated threat for a personal archive — and it is precisely what created the poison pill: phase 6 documents photo `.size` as the size *variant* Telethon selects, i.e. approximate, while an exact-equality gate would delete every completed photo on resume and then raise. The fsync is what licenses the simpler predicate. + +Stray `.part` files are swept at startup, **after** the lock is acquired — the sweep is the destructive operation, so locking after it locks after the damage. + +### Validator: advisory, never raises + +```python +actual, declared = tmp.stat().st_size, media_size(msg) +if declared is None or actual == declared: accept +elif kind(msg) == "photo": accept, log DEBUG # variant size, expected +else: RETRY; on exhaustion -> FAILED +``` + +Documents, video, and audio carry an exact `document.size`, so a mismatch there is real truncation. Photos are the only approximate case. This design is correct **whether or not** phase 1 P5 finds photo sizes exact. + +### Error taxonomy — the failure paths get the same rigor as the happy path + +Three buckets. `FloodWaitError` never appears here: it is owned by `with_flood_retry`, because retries during an active wait are counted as fresh violations. + +```python +TRANSIENT_ERRNO = {EAGAIN, EINTR, EIO, ETIMEDOUT, ECONNRESET} +FATAL_ERRNO = {ENOSPC, EDQUOT, EROFS, EACCES} +MAX_ATTEMPTS, BACKOFF = 3, lambda n: 5 * 2 ** (n - 1) # 5, 10, 20 s + +async def download_one(client, entity, post_dir, msg) -> Result: + target = safe_join(post_dir, derive_filename(msg)) + if target.exists(): + if target.stat().st_size > 0: return SKIPPED + target.unlink() + check_disk(msg) # before the first byte + tmp, refreshed = target.with_name(target.name + ".part"), False + + for attempt in range(1, MAX_ATTEMPTS + 1): + try: + with open(tmp, "wb") as f: + await with_flood_retry(lambda: client.download_media(msg, file=f)) + f.flush(); os.fsync(f.fileno()) + if validate(tmp, msg) is RETRY: + tmp.unlink(missing_ok=True); await sleep(BACKOFF(attempt)); continue + os.replace(tmp, target) + return DOWNLOADED(target) + + # ---- SKIP: this message is unavailable; the run continues ---- + except FileReferenceExpiredError: + tmp.unlink(missing_ok=True) + if refreshed: return FAILED(msg, "file reference expired") + msg = await client.get_messages(entity, ids=msg.id); refreshed = True + if msg is None or msg.media is None: return FAILED(msg, "message deleted") + continue + except (MediaEmptyError, FileIdInvalidError, FilerefUpgradeNeededError): + tmp.unlink(missing_ok=True); return FAILED(msg, "media unavailable or expired") + + # ---- ABORT: the run cannot continue ---- + except (AuthKeyUnregisteredError, AuthKeyDuplicatedError, + SessionRevokedError, UserDeactivatedBanError): + tmp.unlink(missing_ok=True); raise Abort("session invalidated — re-login", 4) + except (ChannelPrivateError, ChatForbiddenError, ChannelInvalidError): + tmp.unlink(missing_ok=True); raise Abort("lost access to the group", 5) + + except OSError as e: # fatal checked FIRST + tmp.unlink(missing_ok=True) + if e.errno in FATAL_ERRNO: raise Abort(f"filesystem: {e}", 3) + if e.errno in TRANSIENT_ERRNO: await sleep(BACKOFF(attempt)); continue + raise Abort(f"unexpected OS error: {e}", 1) + + except (ConnectionError, asyncio.TimeoutError, asyncio.IncompleteReadError, + rpcerrorlist.TimedOutError, rpcerrorlist.RpcCallFailError): + tmp.unlink(missing_ok=True); await sleep(BACKOFF(attempt)); continue + + return FAILED(msg, f"exhausted {MAX_ATTEMPTS} attempts") +``` + +`FileReferenceExpiredError` handling is **not optional** — file references expire in hours, so a 20-hour run will hit it. One `get_messages` refresh is four lines and the highest-value item in this table. + +If phase 1 P4 finds `download_media` rejects a file object, fall back to download-to-path then re-open and fsync before rename. + +### The sink loop — Invariant 2 by construction + +```python +async def run_download(posts, state, sidecar, root): + try: + async for post in posts: + media, results = [m for m in post.messages if m.media], [] # results must be bound + if media: # text-only members download nothing + results = [await download_one(...) for msg in media] + fsync_dir(post_dir) + if post.messages: # may be text-only under --include-text + sidecar.append(post, results); sidecar.fsync() + if wrote_files or (monotonic() - last_commit) > 5.0: + state.commit(post.max_message_id) # only after a COMPLETE post + state.mark_completed() + except Abort as a: + log.error(a.reason); sys.exit(a.code) # cursor already correct + except KeyboardInterrupt: + log.info("interrupted — re-run the same command to resume"); sys.exit(130) + finally: + await client.disconnect() +``` + +**The cursor is never written from an abort path**, so it cannot point past an in-flight post. "Commit the cursor and exit" is deleted as a concept — it was the mechanism by which the disk guard could have violated Invariant 2. + +The 5-second throttle stops 100k filtered messages from causing 100k state writes. Losing an uncommitted sweep-only advance costs a cheap re-sweep with zero downloads. + +Exit codes: `0` ok · `1` unexpected · `2` dry-run won't fit · `3` disk or lock contention · `4` auth · `5` access · `6` flood ceiling (**only reachable when `--max-flood-wait` is explicitly set** — unset means sleep indefinitely) · `7` filter/chat mismatch · `130` interrupt. + +### Disk guard — one mechanism, per file + +```python +def check_disk(msg): + need = (media_size(msg) or 0) + cfg.min_free # default 2 GiB + if shutil.disk_usage(out_dir).free < need: + raise Abort(f"insufficient disk: need {need}, have {free}", 3) +``` + +Replaces all three previous mechanisms. `shutil.disk_usage` is one `statvfs` — microseconds, unmeasurable against multi-second downloads at concurrency 1. The every-50-files poll optimized a cost that does not exist while opening a ~100 GB blind spot: a single Telegram file reaches 2–4 GB, so up to ~100 GB could be written between two polls on a 41 GB-free host. Comparing against *this file's* size closes that structurally. The 2 GiB reserve is principled — it covers one entire unknown-size file at the non-premium upload cap. + +### State file + +```json +{ + "version": 1, + "chat_id": -1001234567890, + "chat_title": "My Group", + "filters": {"types": ["photo","video"], "since": "2026-01-01T00:00:00Z", + "until": null, "max_size": 104857600, "include_text": false}, + "cursor_id": 1042, + "completed_at": null, + "started_at": "2026-08-11T07:00:00Z", + "updated_at": "2026-08-11T07:42:11Z" +} +``` + +- **One cursor field.** `swept_to_id` vs `last_done_id` is a distinction without a difference here: the sweep is a generator driven by the sink at concurrency 1, so it can never run ahead. Two fields would invent a divergence to manage. `cursor_id` means: *every post with `max_message_id <= cursor_id` has been handled under `filters`*. +- **Filters stored verbatim; the sha256 fingerprint is deleted.** A digest is one-way, and two phases require the tool to *name* what changed. Comparing dicts does that for free, with less code than hashing. +- **`counters` deleted.** Derivable from the sidecar and double-counting on every resume (re-processed albums re-increment), which would make phase 7 chase phantom discrepancies. Each run prints its own in-memory session totals at exit. +- **`completed_at`** — set on clean exhaustion, cleared to `null` at every run start. The only way to answer "did my 20-hour export finish?" without guessing. +- Mismatch on `filters` or `chat_id` → refuse, print the diff, exit 7: + ``` + filter mismatch: + types: ['photo','video'] -> ['video'] + max_size: 100MB -> (none) + re-run with the original filters, or --reset-state to start over + ``` + +`--limit` is **not** in the filter set — it changes when we stop, not what a post contains, so including it would make a limited test run poison every later full run. A `--limit 20` run is a resumable partial export whose cursor genuinely reflects completed work. + +### Sidecar + +```json +{"message_id":1043,"post_id":1042,"grouped_id":88,"date":"2026-07-04T12:00:00Z", + "sender_id":777,"sender_name":"Alice","caption":"beach trip","reply_to":1001, + "files":[{"path":"1042/1043_photo.jpg","size":1234567,"declared_size":1234999, + "name":"photo.jpg","mime":"image/jpeg","kind":"photo"}], + "errors":[]} +``` + +Under `--include-text` a text-only message produces the same record shape with `"files":[]` — no folder is created for it. Consumers distinguish media from context by `files` being empty, not by a separate record type. + + + +`size` is bytes on disk; `declared_size` appears only when it differs — which incidentally accumulates empirical data on photo-variant accuracy. Paths are relative to the export root. `sender_name` comes from `msg.sender` **only if already cached** — never an extra `get_entity` call, which on a 100k-message sweep would be 100k extra RPCs and a flood ban. + +**A record is an append-only event: "the files written for this message in this run."** The correct consumer rule is therefore **union over all records for a `message_id`**, not last-write-wins. Last-write-wins was actively wrong — after a filter change it would report a narrower file list than what is on disk. + +Startup repair: scan back from EOF for the last `\n`; if the trailing segment is non-empty and not valid JSON, truncate to that offset and warn. A host crash mid-`write()` otherwise leaves a partial line that breaks every JSONL consumer. + +`--reset-state` **rotates** `messages.jsonl` → `messages-.jsonl` and starts fresh, so the current sidecar always describes exactly one filter regime. It does **not** delete downloaded files — dedupe reclaims them; deleting 30 GB to change a filter is hostile. + +## Related Code Files + +- Create: `src/telegram_exporter/downloader.py`, `state.py`, `sidecar.py` +- Create: `tests/test_download_decisions.py`, `tests/test_state.py` +- Modify: `src/telegram_exporter/cli.py` (wire the download path, `--reset-state`, `--min-free`, `--max-flood-wait`) + +## Implementation Steps + +1. `state.py`: load/create, filter and chat_id comparison with a named diff, atomic save (tmp → fsync → replace → fsync dir), `commit`, `mark_completed` +2. `sidecar.py`: append-only JSONL writer, relative paths, fsync, startup partial-line repair, rotation on `--reset-state` +3. `downloader.py`: `check_disk`, `validate`, `download_one` with the full taxonomy, startup `.part` sweep (after lock acquisition) +4. `downloader.py`: `run_download` per the sketch — sidecar before cursor, cursor unreachable from abort paths +5. Progress output: per-file line with post id, filename, size, running total; session totals at exit including `N files failed (see messages.jsonl)` +6. Write `/title.txt` once on first run +7. Tests per the table below + +## Test Table — `test_download_decisions.py` (tmpdir + fake downloader, no network) + +| Case | Expected | +|---|---| +| target absent | DOWNLOADED | +| target exists, non-empty | SKIPPED, no download call | +| target exists, zero bytes | unlinked, re-downloaded | +| stray `.part` at startup | swept | +| `.part` present and target valid | target wins, `.part` removed | +| `media_size` is `None` | stated policy, asserted | +| `file.name == "../../../etc/passwd"` | lands inside `post_dir` | +| free space < file size + reserve | `Abort(3)` before any write | + +## Success Criteria + +- [ ] Full run downloads every media file into album-aware folders +- [ ] `kill -9` mid-run then re-run: no re-downloads, no gaps, no orphan `.part` +- [ ] Interrupting mid-album re-downloads that album and leaves no duplicates +- [ ] Every row of the test table passes +- [ ] Re-running with changed filters refuses and prints a per-filter diff (exit 7) +- [ ] A different `chat_id` against an existing state file refuses (exit 7) +- [ ] `messages.jsonl` has one line per media message with correct relative paths; a truncated final line is repaired at startup +- [ ] `--reset-state` rotates the sidecar and preserves downloaded files +- [ ] Media with a dead file reference is refreshed once, then recorded in `errors[]` and **skipped** — the run continues past it +- [ ] With `--include-text`, text-only messages appear in the sidecar with `"files":[]` and create no folder +- [ ] Toggling `--include-text` against an existing cursor triggers the filter-mismatch refusal (exit 7) +- [ ] A flood wait logs its duration **and computed wake time**; with no `--max-flood-wait` set, the run sleeps rather than exiting +- [ ] Simulated low disk aborts before writing, with a resumable cursor +- [ ] `completed_at` is set only on clean exhaustion +- [ ] `test_no_concurrency.py` (AST scan) finds no `gather`/`create_task`/`to_thread`/thread pool in `src/` + +## Risk Assessment + +- **Cursor ahead of reality** → permanent silent gaps; the worst failure here because it looks like success. Invariant 2 is structural: `state.commit` is unreachable from every abort path. +- **Right-sized corrupt file after host crash** → fsync before `os.replace`, plus dir fsync before the cursor claims the post durable. This is what makes existence-based dedupe sound. +- **Poison-pill validator** → validator never raises; photos excused; design independent of phase 1 P5's answer. +- **Unhandled exception kills an unattended 20-hour run** → full taxonomy above; transient errors retry with backoff, terminal ones exit cleanly with a correct cursor. +- **Filter/cursor confusion** → verbatim filters, named diff, hard refusal. +- **Disk exhaustion mid-write** → per-file pre-check against this file's size plus a 2 GiB reserve. +- **Concurrent runs** → flock acquired before the `.part` sweep (phase 2). +- **Sidecar misread after a filter change** → union semantics, documented; `--reset-state` rotates. +- **Temptation to parallelize when it feels slow** → flood-wait escalation to multi-hour lockouts. Enforced by an AST test, not by discipline. diff --git a/plans/260811-1411-telegram-group-media-exporter/phase-06-dry-run-estimator.md b/plans/260811-1411-telegram-group-media-exporter/phase-06-dry-run-estimator.md new file mode 100644 index 0000000..f1821b9 --- /dev/null +++ b/plans/260811-1411-telegram-group-media-exporter/phase-06-dry-run-estimator.md @@ -0,0 +1,102 @@ +--- +phase: 6 +title: "Dry-Run Estimator" +status: complete +priority: P2 +dependencies: [5] +effort: "~1h" +--- + +# Phase 6: Dry-Run Estimator + +## Overview + +`--dry-run`: sweep, download nothing, report what a real run would fetch **from where it would actually start**. With 41 GB free and an unknown group size, this is the difference between a planned export and a surprise. + +## Requirements + +- Functional: counts and bytes by media type honoring the same filters *and the same cursor*; disk comparison; non-zero exit when it won't fit +- Non-functional: no `download_media` calls; no writes anywhere, including no creation of the export root; no lock taken; no state mutation + +## Architecture + +`run_estimate()` drives the **same** `iter_posts` generator with the same `Filters` as `run_download()`. That shared generator is the anti-drift property — an estimator with its own traversal would report numbers that don't match reality. There is no `Sink` Protocol: both run modes own a plain `async for`, and the duplication is two lines. + +### The cursor fix + +Dry-run **reads** state (never writes) and starts where the real run would: + +```python +state = State.load(out_dir) # read-only +start = state.cursor_id if state and state.filters == filters else 0 +``` + +Without this, `tg-export --dry-run && tg-export` fails **closed** on every resumed run: a 38 GB group with 30 GB already downloaded and 11 GB free would re-measure the whole 38 GB, exit 2, and refuse to fetch the 8 GB that fits comfortably. The user's only escape would be to bypass the guard they were told to trust. + +Deliberately **not** doing a "scan the export dir and subtract bytes on disk" pass: with a cursor-based sweep everything before the cursor is already excluded, and the residue is a handful of failed files and one partial post. A disk-scan reconciliation is unearned complexity. + +### Report + +``` +Group: My Group (id -1001234567890) +Filters: types=video,photo since=2026-01-01 max-size=100MB +Mode: resume from message 1042 # or "full history (no prior state)" + +Remaining: 1,204 posts / 3,891 files + photos 1,902 files 2.1 GiB (approximate — size variant) + videos 1,944 files 35.4 GiB + documents 45 files 0.7 GiB + unknown size 3 files (not counted) + +Total: 38.2 GiB +Free: 11.0 GiB Reserve: 2.0 GiB +Verdict: SHORT by 29.2 GiB exit 2 +``` + +Unknown-size files get their own line rather than being coerced to zero — an unbounded estimate error must be visible. Photo totals are labelled approximate; document/video/audio are exact. + +Posts and files count **posts with files** — neither empty posts (fully filtered groups, which phase 3 yields to keep the cursor advancing) nor text-only posts under `--include-text` are counted, on both sides of any comparison. Text messages carry no bytes, so the estimate is unaffected by the flag. + + + + +### The verdict is advisory; the per-file guard enforces + +Phase 5's per-file `check_disk` is the sole enforcement mechanism. This removes the previously "optional preflight behind a flag" — a flag that appeared in no flag surface — and the double sweep it implied on huge groups. + +Say this plainly in the README: a `SHORT` verdict does not mean the run is unsafe. It means the run will stop partway with a clean, resumable cursor. + +## Related Code Files + +- Create: `src/telegram_exporter/estimate.py` +- Modify: `src/telegram_exporter/cli.py` (`--dry-run` dispatches to `run_estimate`) + +## Implementation Steps + +1. `EstimateSink` accumulating posts-with-files, files, and bytes by media kind, plus an unknown-size tally +2. Human-readable byte formatting (GiB/MiB, thousands separators) +3. `State.load` in read-only mode; derive `start` and the `Mode:` line +4. Free space via `shutil.disk_usage(out_dir)`; compare `total + reserve` against free +5. Report renderer per the layout; label photo totals approximate +6. `cli.py`: `--dry-run` → `run_estimate`, taking no lock and creating no directories +7. Verify dry-run leaves the filesystem untouched + +## Success Criteria + +- [ ] `--dry-run` completes with zero `download_media` calls and zero bytes written +- [ ] Dry-run does not create the export root, `.export-state.json`, `messages.jsonl`, or `.export.lock` +- [ ] Dry-run takes no lock — it can run while a real export is in progress +- [ ] On a resumed export, `Mode:` shows the cursor and totals cover only remaining work +- [ ] `tg-export --dry-run && tg-export` succeeds on a partially completed export that fits +- [ ] Totals broken down by media kind; unknown-size files on their own line, never as zero +- [ ] Exit `2` when total + reserve exceeds free space; `0` otherwise +- [ ] Filters applied identically to a real run via the shared `iter_posts`, not by duplicated logic +- [ ] `--dry-run --limit 20` vs `--limit 20` from the same start: post and file counts **exactly equal**, bytes within **2%** (photo approximation only) + +## Risk Assessment + +- **Estimator drifts from downloader** → the number users trust becomes a lie. Prevented structurally by the shared `iter_posts`; never fork the traversal. +- **Dry-run mutates state** → a later real run resumes from a cursor nothing downloaded. Read-only by construction; asserted. +- **Dry-run fails closed on resume** → fixed by reading the cursor. This was the defect that made the documented safe idiom unusable. +- **Photo approximation misread as exact** → labelled, plus the 2% tolerance is stated rather than left to judgement. +- **Sweeping a very large history triggers flood waits** → history reads are cheap and covered by `aiter_with_flood_retry`. diff --git a/plans/260811-1411-telegram-group-media-exporter/phase-07-end-to-end-validation.md b/plans/260811-1411-telegram-group-media-exporter/phase-07-end-to-end-validation.md new file mode 100644 index 0000000..e297c29 --- /dev/null +++ b/plans/260811-1411-telegram-group-media-exporter/phase-07-end-to-end-validation.md @@ -0,0 +1,104 @@ +--- +phase: 7 +title: "End-to-End Validation" +status: pending +priority: P1 +dependencies: [6] +effort: "~2h + export runtime" +--- + +# Phase 7: End-to-End Validation + +## Overview + +Prove the tool against a real group, with deliberate interruption. The failure modes that matter — cursor ahead of reality, split albums, flood-wait escalation — are invisible in unit tests and surface only against live data. + +Every step below has a checkable outcome. **No step may pass by escape clause.** + +## Requirements + +- Functional: full export completes, resumes correctly after kill, produces a consistent sidecar +- Non-functional: README documents setup and semantics; no secret or third-party data is committable; the spike's server-side session is revoked + +## Validation Script + +1. **Dry run** — `tg-export --group --out ./exports --dry-run`. Record posts, files, total bytes, verdict. Confirm no export root was created. + +2. **Limited real run** — `--limit 20`. Inspect by hand: albums collapsed into one folder, standalone messages in their own, filenames `{message_id}_{name}`, `title.txt` written, directory named `g`. + +3. **Sidecar cross-check** — scripted, not by eye: + ```bash + python3 - <<'EOF' + # 1. every files[].path in messages.jsonl exists on disk + # 2. every file on disk appears in the UNION of records for its message_id + # 3. no two post folders share a grouped_id <- catches album splitting + EOF + ``` + +4. **Idempotency** — `--reset-state`, then re-run `--limit 20`. Expect **zero downloads, all skips**. + *(Re-running `--limit 20` without resetting would correctly continue to posts 21–40 — the cursor advanced because those 20 posts genuinely completed. Resetting is what isolates the dedupe path, which is the thing under test.)* + +5. **Interruption** — start a full run, `kill -9` after ~50 files. Re-run. Confirm: no surviving `.part`, no re-downloads of completed files, no gaps, the in-flight album complete. + +6. **Network loss** — drop the interface or block Telegram with iptables mid-download for ~30s, then restore. Confirm the transient path retries with backoff and the run continues. `kill -9` alone does not exercise exception propagation or reconnect, so it cannot stand in for this. + +7. **Filter guard** — re-run with `--types video` against the existing state. Confirm it refuses, **names the changed filters**, and exits 7. Then `--reset-state` and confirm it proceeds and rotates `messages.jsonl`. + +8. **Concurrency guard** — start a run; while it holds the lock, start a second against the same `--out`. Confirm exit 3 with pid/host/start-time, and that no file was touched. Confirm a run against a *different* group proceeds. + +9. **Path safety** — covered by `tests/test_paths.py` and `tests/test_download_decisions.py`, which include the hostile-filename case offline. *(The previous "otherwise rely on unit tests and note no live sample existed" escape clause is deleted — it made passing the default.)* Optionally self-send a message named `../../../etc/passwd` to a scratch group for live confirmation. + +10. **Full export** — run to completion. Confirm `completed_at` is set. Reconcile against step 1: post and file counts **exactly equal**; bytes within **2%** (photo approximation only). Any file-count delta is a bug to investigate, never accepted — expected differences are only deleted/expired media, which appear in `errors[]` and the exit summary. + +11. **Flood-wait behavior** — grep logs: each wait slept through once, no retry storm, no restart during an active wait, and every entry carries a computed wake time. Separately confirm `--max-flood-wait 1` exits 6 cleanly with a resumable cursor, since that path is unreachable by default. + +11b. **`--include-text`** — run with the flag on a scratch range. Confirm text-only messages appear with `"files":[]`, create no folders, and that toggling the flag against the existing cursor triggers the exit-7 refusal. + +12. **Memory** — `/usr/bin/time -v tg-export --dry-run` → peak RSS under 200 MB. The album tripwire set is O(albums), not O(1); the criterion states the real bound. + +13. **Revoke the spike session** — Telegram → Settings → Devices → terminate the phase 1 probe session. **Deleting `spike/probe.session` does not revoke server-side authorization**; a live auth key otherwise survives for an account you believe is clean. + +## Related Code Files + +- Modify: `README.md` +- Delete: `spike/` (or leave gitignored) + +## README Must Document + +- Obtaining `api_id`/`api_hash`; the credential policy table (env vs prompt vs getpass; what `.env` may contain) +- First-run login incl. 2FA; session path and why it defaults outside the repo +- **Why the Bot API cannot do this** — the first question any reader will have +- Resume semantics: the cursor means "handled through here *under these filters*", and why changing filters requires `--reset-state` +- Sidecar semantics: records are append-only events; **union over records per `message_id`**, not last-write-wins +- Export directory is `g` with the name in `title.txt`, and why (path safety + surviving group renames) +- Why downloads are sequential — flood-wait escalation, and that `--workers` deliberately does not exist +- `--include-text`: off by default; when set, text-only messages appear in the sidecar with `"files":[]` and no folder. It is part of the stored filter set, so toggling it requires `--reset-state` +- **Flood waits have no ceiling by default** — the tool sleeps as long as Telegram demands, logging the computed wake time each time. A long sleep is a sleep, not a hang; check the log before assuming otherwise. Pass `--max-flood-wait SECONDS` to exit resumably instead + + + +- That the dry-run verdict is **advisory**; the per-file disk check enforces. A `SHORT` verdict means the run stops partway, resumably — not that it is unsafe +- `cryptg` is optional and not installed by default: native code in Telethon's AES path, trusted (if adopted) because the Telethon author maintains it — not because it is "CPU-side only" + +## Success Criteria + +- [ ] Every step 1–13, including 11b, passes with its stated outcome — no step passes by escape clause +- [ ] Full export completes with `completed_at` set; counts reconcile within the stated tolerance +- [ ] Kill-and-resume produces a complete export with no gaps and no duplicates +- [ ] Network-loss interruption recovers without operator intervention +- [ ] Filter guard fires with a named diff and clears with `--reset-state` +- [ ] Concurrency guard exits 3; a different group is unaffected +- [ ] `pytest` green; `pip install -r requirements.txt -e .` from a clean venv yields a working `tg-export` +- [ ] `git status` clean of session files, exports, state, lock, and `.env` +- [ ] Spike session terminated server-side +- [ ] README covers every item listed above + +## Risk Assessment + +- **Group larger than free disk** → step 1 reports it; the per-file guard stops cleanly and resumably. Scope down with `--max-size`/`--types` or export to external storage. +- **Long export interrupted by network loss** → step 6 is the proof; step 5 alone would not be. +- **Estimate/actual discrepancy** → expected sources are deleted/expired media (visible in `errors[]`) and photo approximation. Anything else is a filter or traversal bug — investigate. +- **Album split** → step 3's grouped_id check plus phase 3's runtime tripwire. A split that reaches disk is silent otherwise. +- **Account flood-limited during validation** → keep `--limit` small in steps 2–8; run step 10 once, unattended. +- **Third-party data committed** → the sidecar contains message text and sender names. Export root is gitignored and `assert_not_committable` refuses an un-ignored path inside the repo; verify before any commit. +- **Spike credential left live** → step 13. The most-missed cleanup item, because the local file looks like the whole credential. diff --git a/plans/260811-1411-telegram-group-media-exporter/plan.md b/plans/260811-1411-telegram-group-media-exporter/plan.md new file mode 100644 index 0000000..a146286 --- /dev/null +++ b/plans/260811-1411-telegram-group-media-exporter/plan.md @@ -0,0 +1,328 @@ +--- +title: "Telegram Group Media Exporter" +description: "CLI tool exporting all media from a Telegram group to local disk, grouped by message/album, with resume, filters, and dry-run size estimation." +status: in-progress +priority: P2 +branch: "main" +tags: [telethon, python, cli, mtproto, export] +blockedBy: [] +blocks: [] +created: "2026-08-11T07:22:18.583Z" +createdBy: "ck:plan" +source: skill +--- + +# Telegram Group Media Exporter + +## Overview + +CLI tool `tg-export` — downloads all media from a Telegram group to local disk, one folder per logical post (albums collapsed), with a `messages.jsonl` sidecar preserving message context. Resumable, filterable, with a dry-run size estimate. + +Greenfield repo. Research: [research report](../reports/research-telethon-mtproto-260811-1411-telegram-group-media-exporter-report.md). Reviewed adversarially — see [Red Team Review](#red-team-review); the design below is post-review. + +## Locked Decisions + +Settled before planning — do not re-litigate without new evidence. + +| Decision | Rationale | +|---|---| +| Telethon + Python 3.12, MTProto **user account** | Bot API is structurally incapable: no history access, 20 MB download cap | +| Sequential downloads, **concurrency 1** | Telethon does not parallelize transfers; parallelism accelerates FloodWait escalation (7s → 4h+) | +| Album-aware folders, id = **lowest message id** in the group | User decision; deterministic across re-runs | +| `SQLiteSession` file (not StringSession) | Persistent local filesystem; StringSession solves a containerization problem we don't have | +| `argparse` (not Typer/Click) | One command, ~12 flags, no subcommands | +| `pyproject.toml` + `[project.scripts]` | Current packaging standard | +| Telethon pinned `>=1.41,<2` + `requirements.txt` | Telethon 2.0 is alpha with a renamed API surface; the pin file records the version phase 1 actually verified | + +## Architecture + +Both run modes drive the **same** `iter_posts` generator with the same `Filters`. That shared generator — not any class hierarchy — is what guarantees the dry-run estimate describes what a real run would download. A forked traversal would drift and lie. + +``` +cli.py ──> session.py ──> traversal.py ──┬──> run_download() ──> downloader.py, state.py, sidecar.py + argparse auth, flood sweep + │ + umask wrapper album group └──> run_estimate() ──> estimate.py + logging + filters + locks paths.py (shared, security-critical) +``` + +``` +src/telegram_exporter/ + cli.py argparse, umask, logging, locks, git-safety asserts, dispatch + session.py credentials, session path prep, client, flood retry, entity resolution + traversal.py Post, Filters, media_size(), iter_posts, album-split tripwire + paths.py export_root, post_dir, sanitize, derive_filename, safe_join (pure) + downloader.py download_one, error taxonomy, .part sweep, run_download() + state.py resume cursor only + sidecar.py messages.jsonl append writer only + estimate.py EstimateSink, byte formatting, run_estimate() +tests/ + test_paths.py sanitize table, safe_join, export_root type guard + test_download_decisions.py skip/re-download/.part logic against a tmpdir + test_traversal.py album grouping + split tripwire fires + test_state.py filter mismatch, atomic save + test_lock.py subprocess contention + test_logging.py telethon logger stays >= WARNING + test_no_concurrency.py AST scan rejects gather/create_task +``` + +**No `Sink` Protocol.** Two implementations with one call site and no shared behavior is indirection describing a `for` loop. `run_download()` and `run_estimate()` each own their own `async for`. The anti-drift property lives in the shared `iter_posts` — restated here so nobody re-adds the Protocol believing it was load-bearing. + +**`state.py` and `sidecar.py` are separate modules on purpose.** The correctness-critical part is the *ordering* between them, and that ordering belongs to the caller. Two objects in two modules makes the sequence visible at one call site and makes a "helpful" `commit_and_append()` — which would destroy Invariant 2 — impossible to write without noticing. + +Output layout: + +``` +exports/g-1001234567890/ <- directory name is the chat id, never the group title + title.txt <- human-readable name lives here, as data + 1042/ <- album (shared grouped_id), folder = first member id + 1042_photo.jpg <- filename keyed on message id, not a positional index + 1043_photo.jpg + 1044_video.mp4 + 1047/ <- standalone message + 1047_document.pdf + messages.jsonl + .export-state.json + .export.lock +``` + +**Exactly one untrusted string ever becomes a path component: the filename leaf.** Every other component is an `int` enforced by a type check. The group title is server-supplied and editable by any admin, so it is stored as data and never as a path — which also stops a group rename from orphaning an in-progress export. + +## Three Invariants + +Violating any of these produces silent corruption that looks like success. + +1. **Identity is filter-invariant.** The folder is the lowest message id in the *full* `grouped_id` group; the filename is keyed on `message_id`. Neither depends on which members a filter dropped, so `--types photo` and `--types video` address the same files. +2. **The cursor commits only after a complete post, and never from an abort path.** `state.commit()` is unreachable from every error exit, so the cursor can never point past an in-flight post. +3. **Filters are stored verbatim in the state file.** The cursor means "handled through here *under these filters*". A mismatch refuses the run and names what changed. + +## Acceptance Criteria + +`[x]` = verified offline by the test suite. `[~]` = implemented and unit-tested, +but the end-to-end proof is phase 7 against a live group. + +- [~] `tg-export --group --out ./exports --dry-run` reports remaining posts/files/bytes by media type and a disk verdict, writing nothing — *"writes nothing" and the per-kind breakdown are tested; the live sweep is phase 7 step 1* +- [~] `tg-export --group --out ./exports` downloads all media into album-aware folders +- [~] Killing the run mid-export and re-running resumes with no re-downloads and no gaps — *dedupe, cursor and `.part` sweep are tested; the `kill -9` proof is phase 7 step 5* +- [x] `messages.jsonl` links every file to its message id, post id, date, sender, and caption +- [x] A message whose filename is `../../../etc/passwd` writes inside the post dir, proven by test +- [x] A group titled `..` or `a/b` cannot affect the output path, proven by test — *structurally: no code path exists from title to path component* +- [x] A run aborts before writing the first byte of any file that would not fit, leaving a resumable cursor +- [x] A second concurrent run against the same export root exits 3 without touching any file — *real subprocess contention* +- [x] `FloodWaitError` is slept through once and logged with its computed wake time; no tight retry; a wait beyond an explicit `--max-flood-wait` cap exits cleanly and resumably +- [x] Media that is deleted, expired, or unavailable is recorded and **skipped**, and the run continues +- [x] `--verbose` leaves the `telethon` logger at `WARNING`; `api_hash` and phone absent from captured output +- [x] Session file and every sibling are mode `0600`; export tree `0700` +- [x] `pytest` green (180 tests); `pip install -r requirements.txt -e .` from a clean venv yields a working `tg-export` + +## Phases + +| Phase | Name | Status | +|-------|------|--------| +| 1 | [Verification Spike](./phase-01-verification-spike.md) | Partial — offline probes settled, live probes await credentials | +| 2 | [Skeleton and Session Auth](./phase-02-skeleton-and-session-auth.md) | Complete | +| 3 | [Traversal Grouping and Filters](./phase-03-traversal-grouping-and-filters.md) | Complete | +| 4 | [Naming and Path Safety](./phase-04-naming-and-path-safety.md) | Complete | +| 5 | [Downloader State and Resume](./phase-05-downloader-state-and-resume.md) | Complete | +| 6 | [Dry-Run Estimator](./phase-06-dry-run-estimator.md) | Complete | +| 7 | [End-to-End Validation](./phase-07-end-to-end-validation.md) | Pending — requires a live group | + +**Dependency chain:** 1 → 2 → 3 → 4 → 5 → 6 → 7. Phase 4 is pure and offline-testable, so it can run parallel to 3. Phase 1 gates everything. + +## Open Questions + +1. **Group scale** — unknown volume. Resolved by phase 6 dry-run. +2. **2FA enabled?** — resolved by phase 1 P2. +3. **Group access type** — resolved by phase 1. +4. ~~**Text-only messages in `messages.jsonl`?**~~ **Resolved (validation):** default media-bearing only, plus an `--include-text` flag that records text messages with an empty `files[]`. The flag is part of the stored filter set, since it changes what a post contains. +5. **Deleted/expiring media** — recorded as an `errors[]` entry on that message's sidecar record and counted at exit; the run continues. +6. **Stray `.part` files** — swept at startup, after the lock is acquired. +7. **Unknown-size media (`file.size is None`)** — **included** and warned, counted separately in the estimate rather than as zero. Invert if disk pressure matters more than completeness. +8. **Export dir named `g`, not the group title** — chosen to make path traversal structurally impossible and to survive group renames. Human name lives in `title.txt`. Reversible if you prefer readable directory names, at the cost of both properties. +9. **Should a FAILED file be permanently abandoned?** Raised by the + post-implementation review, and genuinely unsettled by this plan. Today a + failure advances the cursor and is recorded in `errors[]`, which is right for + deleted or expired media but also covers a file that exhausted its three + retries on transient network trouble — that one is never re-attempted, and + `completed_at` is still set. The exit summary names the count. Options if this + matters: re-drive the range with `--reset-state`, or add a `--retry-failed` + pass that reads the sidecar. **Not changed unilaterally — this is a scope + decision.** +10. **Concurrent exports of two groups need two `--session` files**, not just two + `--out` directories, because the session SQLite database is locked too. The + plan claimed per-root locking made concurrent exports work; it does, but only + with that second flag. Documented in the README and in the exit-3 message. + +## Dependencies + +No cross-plan dependencies. + +External: `telethon>=1.41,<2`, pinned in `requirements.txt` to the version phase 1 verified. `cryptg` is **not** installed in v1: it is a native extension sitting in Telethon's AES path, so it sees the auth key and every plaintext byte. It ships an aarch64 wheel (verified — no build step), and the reason to trust it if adopted is that the Telethon author maintains it, not that it is "CPU-side only". Don't add a supply-chain root for a throughput problem that has not been measured. + +## Red Team Review + +### Session — 2026-08-11 + +**Reviewers:** 4 (Security Adversary, Failure Mode Analyst, Assumption Destroyer, Scope & Complexity Critic) +**Findings:** 39 raw → 20 deduplicated → 15 adjudicated (13 accepted, 2 rejected, 4 corrected as overstated) +**Severity breakdown:** 7 Critical, 8 High + +Evidence standard adapted for a greenfield repo: reviewers cited plan-file and research-report lines, since no source code exists to grep. + +| # | Finding | Severity | Disposition | Applied To | +|---|---------|----------|-------------|------------| +| 1 | Photo `.size` documented approximate but used as an exact-equality gate that raises → poison pill | Critical | Accept | Phase 1, 5 | +| 2 | Filename index computed post-filter → breaks dedupe and determinism | Critical | Accept | Phase 3, 4, 5 | +| 3 | Download loop has no `try`; "skip deleted media" structurally impossible | Critical | Accept | Phase 5 | +| 4 | `` is attacker-controlled and unsanitized | Critical | Accept | Phase 4, plan | +| 5 | Flood-wait wrapper cannot protect the sweep | Critical | Accept (re-reasoned) | Phase 2, 3 | +| 6 | `--limit` advances the cursor; phase 7 idempotency step unrunnable | Critical | Accept (fix relocated) | Phase 7 | +| 7 | Phase 1 creates `.session` before any `.gitignore`; sqlite creates it `0644` before the post-hoc `chmod` | Critical | Accept | Phase 1, 2 | +| 8 | Disk guard fires ≤50 files late; "abort before first byte" unimplemented | High | Accept | Phase 5, 6 | +| 9 | Empty-`Post` contract deferred to the implementer; no completion marker | High | Accept | Phase 3, 5 | +| 10 | Dry-run ignores the cursor → `--dry-run && run` fails closed on resume | High | Accept | Phase 6 | +| 11 | No single-instance lock | High | Accept | Phase 2, 5 | +| 12 | `--reset-state` leaves the sidecar → mixed filter generations | High | Accept | Phase 5 | +| 13 | `msg.file.size is None` handled three incompatible ways | High | Accept | Phase 3, 5, 6 | +| 14 | `--verbose` unmutes telethon loggers; 2FA-env policy contradiction; gitignore coupled to default paths | High | Accept (traceback half rejected) | Phase 2 | +| 15 | Album consecutiveness across pagination unverified and untestable as guarded | High | Accept | Phase 1, 3 | + +**Rejected:** +- *"Fold phase 1 into phase 2."* The ordering bug inside it is real and fixed; the merge argument adds no evidence against the gate itself, which exists to settle facts phases 3/5/6 build on. Fixed the duplication instead. +- *"Drop probe P4 (`download_media` return value) — nothing consumes it."* True as written, but finding 1's fix makes the return value the source of truth for `os.replace`. + +**Corrected as overstated or factually wrong** (verified empirically during resolution): +- `cryptg` "builds from sdist on ARM64, executing at install time" — **false**. It ships a `cp312-manylinux_2_28_aarch64` wheel. The plan's *wording* was still wrong; rationale rewritten. +- `--verbose` "leaks credentials via traceback" — **false**. CPython does not print locals in tracebacks. The logger half is real and fixed. +- Session permission race rated Critical — realistically Medium on a single-user host. Fixed anyway: it is one `umask` line. +- "7 modules for 600 lines" — module count is not a defect metric; cohesion is. `estimate.py` kept. The `Sink` Protocol and the `state.py` bundling were the real findings. + +**Net effect:** the resolved design is *smaller* than the reviewed one. Deleted: the `Sink` Protocol, state `counters`, the sha256 filter fingerprint, the every-50-files disk poll, the optional-preflight flag, the positional filename index, and title-based slugging. + +### Whole-Plan Consistency Sweep +- Files reread: plan.md, phase-01 … phase-07 +- Decision deltas checked: 15 +- Reconciled stale references: filename scheme (ph3/4/5), state schema (ph5/6/7), module names `naming.py`→`paths.py` (plan/ph4/ph5), flag surface (ph2/3/5/6), dedupe predicate (ph5/7), export dir naming (plan/ph4/ph5/ph7), disk guard (plan/ph5/ph6/ph7) +- Unresolved contradictions: 0 + +## Implementation Log + +### Session 1 — 2026-08-12/13 — phases 2–6 + +Phases 2–6 implemented and green: **180 tests**, all offline against synthetic +message stubs. `pip install -r requirements.txt -e .` from a clean venv yields a +working `tg-export` on Telethon **1.44.0** (aarch64 wheel). + +Phase 1 could not be run — no Telegram credentials exist in this environment — +but most of what it gates was settled without them, recorded in the +[spike findings report](../reports/spike-260811-1411-telethon-api-verification-report.md): + +- **P1 CONFIRMED from Telethon's source**, not from a sample: under `reverse=True` + the iterator does `offset_id += 1` before the first request and again per page, + and `min_id` is not consulted in the reverse branch of `_message_in_range`. So + `offset_id` is an exclusive lower bound, the cursor stores the last handled id, + and `iter_posts` carries a defensive `msg.id <= after_id` guard regardless. +- All 15 error classes in phase 5's taxonomy exist in 1.44.0. A missing one would + have been an import-time crash. +- P2, P3, P5 and P6 remain live-only. The design was already answer-independent + for all four, which is why implementation did not wait on them. + +`spike/probe_telethon_api.py` is written and ready to run. + +#### Module deviation + +`media.py` was added, holding `media_kind` / `media_size` / `file_stem` / +`file_ext`. The plan put `media_size` in `traversal.py`, but `paths.py` needs kind +and extension for `derive_filename`, and either direction of import between those +two would have been a cycle or a second copy of the rule. One owner, two +consumers, no cycle. + +One behavioral consequence worth recording: `media_kind` returns `None` when +`msg.web_preview` is set. Telethon's `Message.photo` and `.document` also return +the *web preview's* media, so classifying without that guard would have made every +message containing a URL look like an attachment — inflating counts and +downloading link-preview thumbnails. + +#### Defects found and fixed during implementation + +| # | Defect | Consequence had it shipped | +|---|--------|---------------------------| +| 1 | The 5-second commit throttle never flushed the tail | A run ending on a short stretch of filtered or text-only posts set `completed_at` against a stale cursor; the next run re-swept that tail to download nothing | +| 2 | `git check-ignore` reports `exports` as *not ignored* until the directory exists | The export root would be refused on the first run and accepted on every run after it. Fixed with a directory-form retry, gated to `is_dir=True` so it can never excuse a session *file* | +| 3 | `file_ext` capped extensions at 16 characters, not bytes | A multibyte suffix pushed the filename past ext4's 255-byte limit | + +#### Corrections to the plan's own sketches + +- **`except` ordering.** The plan checked `OSError` before the network clause, but + `ConnectionError` *is* an `OSError`, and asyncio's `ConnectionResetError` can + carry `errno=None` — which the errno test would have routed to "unexpected OS + error" and aborted a healthy run. Network clause first, filesystem second. +- **Lock contention exit code.** `raise SystemExit(f"busy: ...")` exits **1**, not + 3, so it would have failed its own acceptance criterion. Raises `Abort(reason, 3)`. +- **Exit-code ownership.** `run_download` lets `Abort` and `KeyboardInterrupt` + propagate to `cli.main()` rather than calling `sys.exit` mid-loop, and does not + call `disconnect()` — the `connected_client` context manager owns that. +- **Bare asserts.** The ascending-sweep and album-buffer tripwires are now + `SweepOrderError`, not `assert`: `python -O` strips assertions, and these guard + the same silent corruption as the `AlbumSplitError` beside them. + +### Red Team Review — Session 2 — 2026-08-13 (post-implementation) + +Independent `code-reviewer` pass over the implementation against this plan. +**Verdict: DONE_WITH_CONCERNS.** Invariants 1 and 2 verified to hold under trace +and probe; dry-run and real-run agree exactly on posts and files for +`limit ∈ {None, 0, 1, 2}`. All eight declared deviations were assessed sound and +kept. Two reachable defects broke stated guarantees: + +| # | Finding | Severity | Disposition | +|---|---------|----------|-------------| +| H1 | `--types " "` (or `,`) builds an empty kind set that drops every media kind, but `to_state()` serialized it as `None` — i.e. *no filter* | Critical | **Fixed.** Refused in `build_filters`; `to_state` made lossless as a second line of defense | +| H2 | The filename **extension** bypassed `sanitize` entirely | Critical | **Fixed.** Routed through `sanitize_ext`; plus `download_one` now records a path failure as FAILED instead of letting it escape | +| M1 | `validate` excused *any* photo size mismatch, including 0 bytes | High | **Fixed.** Zero bytes is refused for every kind | +| M2 | A FAILED file advances the cursor and is never retried | Medium | **Documented, not changed** — the plan settles this ("recorded and skipped, the run continues"). See open question 9 | +| M3 | `ServerError` (parent of `RpcCallFailError`) and the `telethon.errors.common` set escaped the taxonomy | High | **Fixed.** Both added to the retry tuple | +| M4 | `OSError` from the sidecar or state writes bypassed the exit-code contract | Medium | **Fixed.** `cli.main` maps `FATAL_ERRNO` → 3 | +| M5 | `--max-size 0` printed as `(none)`; a negative `--limit` made the two run modes disagree | Low | **Fixed.** `is not None` throughout; negative `--limit` rejected at the boundary | +| M6 | `Config.max_flood_wait` was dead | Low | **Fixed.** Removed | + +H1 was the worst of these: it is silent *total* data loss, reachable from +`--types "$UNSET_VAR"`, and it defeats Invariant 3 — the very guard meant to catch +it. H2 was a permanent denial of service from one hostile filename, since the +cursor could not advance past the message that raised. + +Also fixed from the review's lower-severity notes: duplicated `_fsync_dir` +(now `state.fsync_dir`, one owner), the session file being created before the git +check rather than after, dead code in `human_bytes` and `EstimateSink.add`, and a +dry-run `Mode:` line that silently measured from zero when the stored filters +differed — it now says a real run would refuse. + +Two criteria the review found untested are now tested: export-tree and file modes +under the process umask, and control characters in the *extension*. + +## Validation Log + +### Session 1 — 2026-08-11 + +Verification pass skipped per the workflow guard: `## Red Team Review` above already carries evidence-backed verification, and zero `[UNVERIFIED]` tags remain across all eight plan files. + +**Questions asked:** 5 (4 + 1 disambiguation) + +| # | Decision point | Answer | Effect | +|---|---|---|---| +| 1 | Export directory naming | `g` + `title.txt` | **Confirms** the red-team fix. No change. | +| 2 | Media with no reported size | Download + warn | **Confirms** the resolved design. No change. | +| 3 | Sidecar scope | Add an `--include-text` flag | **Change** — see below | +| 4 | Disk reserve | 2 GiB | **Confirms.** No change. | +| 5 | Flood-wait ceiling | **No ceiling by default**; `--max-flood-wait N` caps it | **Change** — reverses the plan's prior default | + +**Change 1 — `--include-text`.** Default remains media-bearing only. With the flag, text-only messages are recorded in the sidecar with an empty `files[]`. It joins the stored filter set because it changes what a post contains: running without it and then with it against the same cursor would otherwise silently skip the text records. Text-only messages never create a post folder and never affect `posts_with_files` counting, so the dry-run/real-run reconciliation is unaffected. + +**Change 2 — flood-wait default inverted.** Previously the tool exited resumably after a wait beyond 1 hour. Now it sleeps indefinitely by default, and `--max-flood-wait N` imposes a ceiling. + +The trade-off was presented and the user chose unattended completion over fail-fast: a badly throttled account now looks like a hang rather than an exit. Accepted with one mitigation — every flood wait logs at WARNING with its duration **and computed wake time**, so a long sleep is legible as a sleep. Exit code 6 is retained for the capped case. + +### Whole-Plan Consistency Sweep — Validation Session 1 +- Files reread: plan.md, phase-01 … phase-07 +- Decision deltas checked: 2 +- Reconciled: flag surface (plan/ph2/ph3/ph5), filter set membership of `--include-text` (ph3/ph5), sidecar record shape for text messages (ph5), `posts_with_files` counting (ph3/ph6), flood-wait defaults and exit code 6 (plan/ph2/ph5), README items (ph7) +- Unresolved contradictions: 0 diff --git a/plans/reports/research-telethon-mtproto-260811-1411-telegram-group-media-exporter-report.md b/plans/reports/research-telethon-mtproto-260811-1411-telegram-group-media-exporter-report.md new file mode 100644 index 0000000..ea989ae --- /dev/null +++ b/plans/reports/research-telethon-mtproto-260811-1411-telegram-group-media-exporter-report.md @@ -0,0 +1,325 @@ +# Research Report: Telegram Group Media Exporter (Telethon / Python) + +**Conducted:** 2026-08-11 14:11 (Asia/Saigon) +**Target env:** Linux ARM64 (aarch64), headless, Python 3.12.3, ffmpeg 6.1.1, 41 GB free disk +**Stack (confirmed by user, not re-evaluated):** Telethon + Python, album-aware folder layout +**Scope in v1:** resume+dedupe, `messages.jsonl` sidecar, type/date/size filters, dry-run size estimate + +--- + +## Table of Contents + +1. [Executive Summary](#executive-summary) +2. [Methodology](#methodology) +3. [Findings](#findings) + - [1. Auth & Session Persistence](#1-auth--session-persistence) + - [2. History Traversal & Resume](#2-history-traversal--resume) + - [3. Album Grouping via grouped_id](#3-album-grouping-via-grouped_id) + - [4. Media Download & Filenames](#4-media-download--filenames) + - [5. Flood Waits & Concurrency](#5-flood-waits--concurrency) + - [6. Dry-Run Size Estimation](#6-dry-run-size-estimation) + - [7. Packaging & CLI](#7-packaging--cli) +4. [Proposed Architecture](#proposed-architecture) +5. [Common Pitfalls](#common-pitfalls) +6. [Confidence Ledger](#confidence-ledger) +7. [Next Steps](#next-steps) +8. [Unresolved Questions](#unresolved-questions) +9. [References](#references) + +--- + +## Executive Summary + +Three findings drive the whole design. + +**Concurrency is the wrong instinct.** Telethon does not parallelize file downloads internally, and the docs/issues are explicit that parallel downloads only make `FloodWaitError` arrive *sooner*. Worse, retrying during an active flood wait is itself counted as a violation — a 7s wait can escalate past 4 hours within an hour of retry-looping. **Default concurrency must be 1**, with explicit flood-wait sleeping. This kills any worker-pool design before it is written. + +**Telethon has no byte-level download resume.** `download_media` writes from offset 0 every call. Resumability must therefore be built at *message granularity*: download to a `.part` file, `os.replace()` atomically on completion, and treat any surviving `.part` as incomplete garbage to be re-fetched. Combined with an oldest→newest sweep (`reverse=True`) and a persisted last-completed message id, this gives a monotonic, crash-safe cursor. + +**Albums need buffering, not lookups.** An album is N separate messages sharing a `grouped_id`; there is no "fetch the album" API. The common StackOverflow pattern (scan ±10 message ids around a post) is for the single-post case and is wasteful here. Since we sweep every message anyway, group by buffering consecutive same-`grouped_id` messages and flushing when the id changes. + +--- + +## Methodology + +- Sources consulted: 5 research passes (1 doc fetch, 4 web searches) → ~20 distinct sources +- Gemini CLI: **absent on this host**, fell back to WebSearch per skill config +- Primary authority: `docs.telethon.dev` (1.44.0 stable) + LonamiWebs/Telethon GitHub issues +- Key terms: `grouped_id album`, `FloodWaitError bulk download`, `StringSession vs SQLiteSession`, `iter_messages reverse offset_id`, `pyproject.toml CLI 2026` +- **Limitation:** `docs.telethon.dev/en/stable/modules/client.html` is a single very large page; the fetch truncated before the full `iter_messages` and `download_media` signatures. Items depending on it are flagged in the [Confidence Ledger](#confidence-ledger). + +--- + +## Findings + +### 1. Auth & Session Persistence + +Bot API is structurally disqualified (no history access; 20 MB download cap) — MTProto user account is the only path. `api_id`/`api_hash` from my.telegram.org. + +| | SQLiteSession (default) | StringSession | +|---|---|---| +| Storage | on-disk `.session` SQLite DB | in-memory, serializable to base64 string | +| Best for | long-lived local tool | serverless / env-var-injected secrets | +| Gotcha | **must call `disconnect()` before exit** or state is not flushed | must persist the string yourself | + +**Recommendation: SQLiteSession file**, `chmod 600`, path configurable, `.gitignore`d. This is a localhost tool with a persistent filesystem — StringSession solves a containerization problem we do not have (YAGNI). One-time interactive login (phone + code), every later run unattended. + +Security, verbatim from the docs: *"Never give the saved session file to anyone, since they would gain instant access to all your messages and contacts."* Treat `.session` as a credential — never commit, never log its path contents. + +**2FA is unresolved by the sources.** The searched docs did not cover the 2FA password flow. Telethon's `client.start(phone=..., password=...)` and `SessionPasswordNeededError` are the expected mechanism — **verify at implementation time**, and never accept the 2FA password as a CLI argument (it lands in shell history); prompt interactively or read from env. + +### 2. History Traversal & Resume + +`iter_messages(entity, ...)` with parameters `limit`, `offset_id`, `min_id`, `max_id`, `offset_date`, `filter`, `reverse` — all confirmed present, full semantics truncated in the fetch. + +**Design: sweep oldest→newest with `reverse=True`.** Rationale: message ids increase monotonically over time, so "highest fully-completed id" is a single-integer resume cursor that is always correct, even if the run is killed mid-album. A newest-first sweep would need an interval set instead. + +```python +# resume: continue after the last fully-completed message +async for msg in client.iter_messages(entity, reverse=True, offset_id=state.last_done_id): + ... +``` + +**Footgun to verify:** `offset_id` semantics *invert* with `reverse`. Newest-first, it is an upper bound (start below this id); with `reverse=True` it acts as a lower bound (start above this id). Confirm empirically with a 3-message probe before trusting the resume path — an off-by-one here silently re-downloads or silently skips. + +Pagination is handled internally (~100 messages/request); no manual paging needed. + +### 3. Album Grouping via `grouped_id` + +Confirmed: *"Albums are a way to send multiple photos or videos as separate messages with the same grouped identifier. It is possible to know that it's a grouped message thanks to `grouped_id`."* + +Album members arrive as **consecutive** messages in a chronological sweep. Grouping algorithm: + +``` +buffer = [] +for msg in sweep(reverse=True): + if msg.grouped_id is None: + flush(buffer); emit_single(msg) + elif buffer and msg.grouped_id != buffer[0].grouped_id: + flush(buffer); buffer = [msg] + else: + buffer.append(msg) +flush(buffer) # end of history +``` + +**Folder id = the lowest message id in the group** (first member, since we sweep ascending). Deterministic and resume-stable: a re-run recomputes the same folder name. + +**Resume interacts with albums.** Only advance `last_done_id` once *every* member of a group has landed — otherwise a crash mid-album leaves a half-folder that resume skips past. Simplest correct rule: commit the cursor at group-flush boundaries, not per file. + +Do **not** use the "iterate ±N ids around the post" pattern from issue #3788 — that solves the single-known-post case, and duplicates work in a full sweep. + +### 4. Media Download & Filenames + +Confirmed chunked download primitive: + +```python +async download_file( + input_location: hints.FileLike, + file: hints.OutFileLike = None, + *, + part_size_kb: float = None, + file_size: int = None, + progress_callback: hints.ProgressCallback = None, + dc_id: int = None, + key: bytes = None, + iv: bytes = None +) -> bytes | None +``` + +For our purposes the higher-level `download_media(message, file=path, progress_callback=...)` is correct — it wraps chunking and DC routing. `part_size_kb` tuning is a micro-optimization; skip it (YAGNI). + +**Filename/extension derivation:** `message.file` exposes `.name`, `.ext`, `.mime_type`, `.size`. Preference order: +1. `DocumentAttributeFilename` via `message.file.name` — real filename for documents +2. `message.file.ext` + mime type — for photos/voice/round video, which carry no filename +3. Fallback `bin` + +**Sanitize aggressively.** Filenames are attacker-controlled strings from arbitrary group members: strip `/`, `..`, NUL, leading dots, control chars, and cap length. Always join under the target dir and assert the resolved path stays inside it. A crafted `../../../.ssh/authorized_keys` is a real, cheap attack on a naive exporter. + +**No byte-level resume** (see Executive Summary). Pattern: + +``` +tmp = folder / f"{idx}_{name}.part" +download_media(msg, file=tmp) +os.replace(tmp, folder / f"{idx}_{name}") # atomic on same filesystem +``` + +**Dedupe check:** skip when the final file exists *and* its size equals `message.file.size`. Size-match is cheap and catches truncation; hashing every file is wasted I/O at multi-GB scale. + +### 5. Flood Waits & Concurrency + +The single most decision-relevant finding. + +- `client.flood_sleep_threshold` (default **60s**): Telethon auto-sleeps on flood waits ≤ threshold, raises `FloodWaitError` above it. +- *"The library does not download or upload files in parallel"* — no internal parallelism to inherit. +- *"The limiting factor in the long run are FloodWaitError, and using parallel download or uploads only makes them occur sooner."* +- **Escalation is real:** *"a wait that starts at 7 seconds can escalate past four hours within less than an hour of repeated restarts, because every retry during an active FloodWait is itself treated as another violation."* + +**Design rules:** +1. Sequential downloads. Default concurrency **1**. Do not ship a `--workers` flag in v1 — it is a footgun that trades a small speedup for hour-long lockouts. +2. Raise `flood_sleep_threshold` (e.g. 120s) so routine small waits are absorbed silently. +3. Catch `FloodWaitError` explicitly, `await asyncio.sleep(e.seconds + jitter)`, then continue. **Never** tight-retry, never restart the process to "get around" a wait. +4. Log every flood wait with its duration — it is the primary signal for whether the run is healthy. + +### 6. Dry-Run Size Estimation + +`message.file.size` gives bytes without fetching content. Dry-run = full `iter_messages` sweep with zero `download_media` calls: cheap (~100 messages/request, no file-DC traffic), and the flood-wait risk is low because history reads are far cheaper than file reads. + +Output: total media count, total bytes, breakdown by type, and **free-disk delta**. With 41 GB free on `/config`, an export that does not fit should fail loudly *before* the first byte, not at 97%. + +Caveat: for photos, `.size` reflects the size variant Telethon selects (largest by default); treat photo totals as a close estimate, document sizes as exact. + +### 7. Packaging & CLI + +Consensus from the packaging guides: `argparse` for simple tools with no dependencies, Click for complex multi-command CLIs, Typer for type-hint ergonomics (built on Click). + +**Recommendation: `argparse`.** One command, ~8 flags, no subcommands. Adding Typer means +2 transitive deps to save perhaps 25 lines. YAGNI/KISS. If subcommands ever appear, Typer is a contained swap. + +Packaging: `pyproject.toml` with `[project.scripts]` entry point (`tg-export = "telegram_exporter.cli:main"`) — the current standard, declarative and static. + +**`uv` is not installed on this host.** Options: `pip install -e .` inside a venv (works today, zero setup), or install `uv` first. The venv path is fine; nothing here needs uv. + +--- + +## Proposed Architecture + +``` + ┌──────────────┐ + │ cli.py │ argparse: --group --out --dry-run + │ │ --since/--until --types --max-size + └──────┬───────┘ + │ + ┌──────▼───────┐ + │ client.py │ auth, SQLiteSession, flood-wait wrapper + └──────┬───────┘ + │ + ┌───────────────────▼────────────────────┐ + │ sweep.py │ + │ iter_messages(reverse=True, │ + │ offset_id=cursor) │ + └───────────────────┬────────────────────┘ + │ messages, ascending + ┌───────────────────▼────────────────────┐ + │ grouper.py buffer by grouped_id │ + └───────────────────┬────────────────────┘ + │ logical posts + ┌─────────────┴─────────────┐ + │ │ + ┌────────▼────────┐ ┌────────▼────────┐ + │ estimator.py │ │ downloader.py │ + │ sum .file.size │ │ .part → rename │ + │ (dry-run only) │ │ size-match skip │ + └─────────────────┘ └────────┬────────┘ + │ + ┌─────────────▼─────────────┐ + │ state.py cursor + jsonl │ + │ commit at group boundary │ + └───────────────────────────┘ +``` + +Output layout (album-aware, per user decision): + +``` +exports// + 1042/ <- album, grouped_id shared, folder = first msg id + 0_photo.jpg + 1_photo.jpg + 2_video.mp4 + 1047/ <- standalone message + 0_document.pdf + messages.jsonl <- id, date, sender, caption, grouped_id, reply_to, files[] + .export-state.json <- last_done_id, counters +``` + +--- + +## Common Pitfalls + +| Pitfall | Consequence | Mitigation | +|---|---|---| +| Parallel downloads to "go faster" | Flood waits arrive sooner, escalate to hours | Sequential, concurrency 1 | +| Retrying during an active flood wait | 7s wait → 4h+ lockout | Sleep the full `e.seconds`, never restart to bypass | +| Forgetting `disconnect()` | SQLite session not flushed, re-login next run | `try/finally` or async context manager | +| Unsanitized `DocumentAttributeFilename` | Path traversal outside the export dir | Sanitize + assert resolved path is inside target | +| Advancing the resume cursor per file | Crash mid-album → permanently half-downloaded folder | Commit cursor only at group flush | +| Assuming `download_media` resumes | Silent truncated files | `.part` + atomic rename; treat stray `.part` as garbage | +| `offset_id` semantics under `reverse=True` | Silent skip or full re-download | Verify empirically with a 3-message probe | +| Committing the `.session` file | Full account takeover | `.gitignore`, `chmod 600` | +| No pre-flight size check | Fills 41 GB disk mid-run | Dry-run + free-space assertion | + +--- + +## Confidence Ledger + +Honest separation of what the sources confirmed from what needs checking in code. + +**Verified by sources:** +- `flood_sleep_threshold` default 60s, auto-sleep ≤ threshold +- Telethon does not parallelize file transfers; parallelism accelerates flood waits +- Flood-wait escalation from retries during an active wait +- `download_file` full signature incl. `part_size_kb`, `progress_callback`, `dc_id` +- `grouped_id` identifies album membership; albums are separate messages +- SQLiteSession vs StringSession trade-offs; `disconnect()` flush requirement +- `iter_messages` accepts `reverse`, `offset_id`, `min_id`, `max_id`, `offset_date`, `filter`, `limit` +- `[project.scripts]` in `pyproject.toml` as the entry-point standard + +**Not confirmed — verify during implementation:** +- Exact `offset_id` direction under `reverse=True` (**highest-risk item**; breaks resume silently) +- 2FA / `SessionPasswordNeededError` flow — sources did not cover it +- `message.file` attribute surface (`.name`/`.ext`/`.size`/`.mime_type`) — strongly indicated by the custom-types docs, not read verbatim +- Whether `download_media` returns the final path string in all media branches +- Photo `.size` accuracy vs the actually-downloaded variant + +--- + +## Next Steps + +1. **Spike first, build second** (~30 min): a throwaway script that logs in, iterates 20 messages of a real group, prints `id / grouped_id / file.name / file.size`, and downloads one album. Settles all four unverified items above at once. +2. Run `/ck:plan` to phase the implementation against the architecture sketch. +3. Phase order: auth+session → sweep+resume → grouper → downloader → jsonl sidecar → filters → dry-run estimator. +4. Add `.session`, `exports/`, `.export-state.json` to `.gitignore` **before** the first login. + +--- + +## Unresolved Questions + +1. **Group scale?** Unknown message/media count. If the group is >40 GB the 41 GB free disk is the binding constraint and dry-run becomes mandatory before any real run. +2. **2FA enabled on the account?** Changes the login flow and whether an interactive prompt is required on first run. +3. **Group access type** — public/private, and is the account already a member? Private groups need the account joined; `get_entity` resolution differs for public username vs invite-link groups. +4. **Should non-media messages appear in `messages.jsonl`?** Text-only messages give context to captions and replies, but inflate the sidecar. Default assumption: media-bearing messages only. +5. **Deleted/expiring media and service messages** — skip silently or record as gaps in the sidecar? +6. **Retention of `.part` files** on a failed run — auto-clean at startup, or leave for inspection? + +--- + +## References + +### Official Documentation +- [Client class — Telethon 1.44.0](https://docs.telethon.dev/en/stable/modules/client.html) — `download_file` signature, `iter_messages` params +- [RPC Errors — Telethon 1.44.0](https://docs.telethon.dev/en/stable/concepts/errors.html) — FloodWaitError semantics +- [FAQ — Telethon 1.44.0](https://docs.telethon.dev/en/stable/quick-references/faq.html) — no-parallel-transfer statement +- [Session Files — Telethon 1.44.0](https://docs.telethon.dev/en/stable/concepts/sessions.html) — SQLite vs String sessions +- [Sessions module — Telethon 1.44.0](https://docs.telethon.dev/en/stable/modules/sessions.html) +- [Custom package (types) — Telethon 1.44.0](https://docs.telethon.dev/en/stable/modules/custom.html) — `Message.file` wrapper +- [Telethon 1.44.0 full PDF](https://media.readthedocs.org/pdf/telethon/stable/telethon.pdf) — offline reference for the truncated pages +- [Creating and packaging command-line tools — PyPA](https://packaging.python.org/en/latest/guides/creating-command-line-tools/) +- [Writing your pyproject.toml — PyPA](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) + +### Issues & Community +- [Download all media in a post by grouped_id — Telethon #3788](https://github.com/LonamiWebs/Telethon/issues/3788) +- [Detect a grouped message (album) — Telethon #1216](https://github.com/LonamiWebs/Telethon/issues/1216) +- [FloodWaitError in client.download_media — Telethon #1426](https://github.com/LonamiWebs/Telethon/issues/1426) +- [FloodWaitError during document download — Telethon #206](https://github.com/LonamiWebs/Telethon/issues/206) +- [Converting StringSession to SQLiteSession — Telethon #3660](https://github.com/LonamiWebs/Telethon/issues/3660) +- [Telethon — DeepWiki](https://deepwiki.com/LonamiWebs/Telethon) + +### Further Reading +- [Fix Telegram FloodWait Error Fast](https://membertel.com/blog/how-to-fix-telegram-floodwait-error-fast/) — flood-wait escalation behavior +- [telebackup — multi-connection Telegram downloader](https://github.com/xwc9527/telebackup) — prior art for parallel/checkpoint downloading; **contrary to our sequential decision**, review only if throughput becomes a proven problem +- [Packaging Python CLI apps the modern way with uv](https://thisdavej.com/packaging-python-command-line-apps-the-modern-way-with-uv/) +- [Building Python CLI tools: Click, Typer, argparse](https://inventivehq.com/blog/python-cli-tools-guide) + +### Glossary +- **MTProto** — Telegram's native protocol; full user-account API access (vs the restricted Bot API) +- **DC** — Telegram data center; large files may live on a DC other than the session's home DC +- **`grouped_id`** — shared identifier marking messages that belong to one album +- **FloodWait** — server-imposed cooldown after too-rapid requests; escalates on retry diff --git a/plans/reports/spike-260811-1411-telethon-api-verification-report.md b/plans/reports/spike-260811-1411-telethon-api-verification-report.md new file mode 100644 index 0000000..b883beb --- /dev/null +++ b/plans/reports/spike-260811-1411-telethon-api-verification-report.md @@ -0,0 +1,139 @@ +--- +title: "Phase 1 — Telethon API Verification Findings" +phase: 1 +status: partial +telethon_version: "1.44.0" +created: "2026-08-12" +--- + +# Phase 1 — Telethon API Verification Findings + +Two sources of evidence appear below and are labelled separately: + +- **SOURCE** — settled offline by reading the installed `telethon` 1.44.0 package. + Deterministic, and stronger than a single live observation for questions about + parameter semantics, because it shows the code path rather than one sample of + its behavior. +- **LIVE** — requires `TG_API_ID`/`TG_API_HASH`, an interactive login, and a real + group. **Not yet run.** Run `spike/probe_telethon_api.py` and paste its output + into the corresponding section. + +Environment: Python 3.12.3, aarch64, Telethon **1.44.0** (wheel, no build step), +pinned in `requirements.txt`. + +## Confidence ledger + +| Probe | Question | Status | Answer | +|---|---|---|---| +| P1 | `offset_id` direction/inclusivity under `reverse=True` | **CONFIRMED (SOURCE)** | Exclusive lower bound: results satisfy `id > offset_id` | +| P2 | 2FA / login flow | **PENDING (LIVE)** | — | +| P3 | `message.file` surface; which kinds lack `.size` | **PENDING (LIVE)** | — | +| P4 | `download_media` return value; file-object support | **PARTIAL (SOURCE)** | File objects are accepted; return value pending live confirmation | +| P5 | photo declared `.size` vs bytes on disk | **PENDING (LIVE)** | — | +| P6 | album adjacency across a pagination boundary | **PENDING (LIVE)** | Runtime tripwire implemented regardless | +| — | error classes in phase 5's taxonomy exist | **CONFIRMED (SOURCE)** | All 15 present | + +## P1 — `offset_id` under `reverse=True` — CONFIRMED (SOURCE) + +`telethon/client/messages.py`, `_MessagesIter._init`: + +```python +if self.reverse: + if offset_id: + offset_id += 1 + elif not offset_date: + offset_id = 1 +``` + +and `_update_offset`, run after every page: + +```python +self.request.offset_id = last_message.id +if self.reverse: + self.request.offset_id += 1 # "We want to skip the one we already have" +``` + +**`offset_id` is an exclusive lower bound under `reverse=True`.** `offset_id=N` +yields messages with `id > N`; `N` itself is excluded. `offset_id=0` starts from +the beginning of history. + +`min_id` was compared as the plan asked: `_message_in_range`'s `reverse` branch +tests `message.id <= self.last_id or message.id >= self.max_id` and **does not +consult `min_id` at all**, so `offset_id` is the correct parameter for a +chronological sweep, not `min_id`. + +**Consequence, implemented:** the cursor stores the **last handled id** and is +passed as `offset_id`. `iter_posts` additionally carries an explicit +`if msg.id <= after_id: continue` guard, so the sweep stays correct even if this +internal detail changes in a future Telethon release — +`src/telegram_exporter/traversal.py`. + +## Error-class inventory — CONFIRMED (SOURCE) + +Every exception phase 5's taxonomy catches exists in 1.44.0 (checked against +`telethon.errors` and `telethon.errors.rpcerrorlist`). A missing name would have +been an import-time crash on the first run: + +`FloodWaitError`, `FileReferenceExpiredError`, `MediaEmptyError`, +`FileIdInvalidError`, `FilerefUpgradeNeededError`, `AuthKeyUnregisteredError`, +`AuthKeyDuplicatedError`, `SessionRevokedError`, `UserDeactivatedBanError`, +`ChannelPrivateError`, `ChatForbiddenError`, `ChannelInvalidError`, +`TimedOutError`, `RpcCallFailError`, `SessionPasswordNeededError`. + +Constructor signature for all of them is `(request)`, with `FloodWaitError` +taking `(request, capture=0)` where `capture` becomes `.seconds`. The test suite +constructs them this way. + +## P4 — `download_media` file-object support — PARTIAL (SOURCE) + +`Message.file` wraps `photo or document` and is `None` for other media types +(polls, games, none), which is why `media.media_kind()` classifies from the media +type rather than from `.file`. + +Telethon accepts a stream for `file=`, so phase 5 holds the fd and `fsync`s it +before `os.replace`. The plan's fallback — download to a path, then re-open and +fsync — is not needed unless the live probe contradicts this. + +**Still pending (LIVE):** the exact return type/value of +`download_media(msg, file=)`. + +## P3 / P5 note — why the implementation does not wait on these + +Both remaining questions are about *sizes*, and the resolved design is +answer-independent by construction: + +- `media.media_size()` returns `int | None`, and each of the four consumers has a + stated rule for `None` (include+warn / accept / separate tally / treat as 0). + Tested in `tests/test_traversal.py`. +- The download validator never raises: a photo mismatch is accepted as a size + variant, and a document/video/audio mismatch retries then reports FAILED. + Tested in `tests/test_download_decisions.py`. + +P3's answer improves the `--max-size` warning's accuracy; P5's improves the +dry-run label and phase 7's reconciliation tolerance. Neither can invalidate code +that already treats `None` as a first-class case. + +## P2 / P6 — PENDING (LIVE) + +- **P2** — record whether `SessionPasswordNeededError` is raised, and confirm a + second run is non-interactive. The 2FA password is `getpass`-only, enforced by + `tests/test_logging.py`. +- **P6** — the probe requires ≥250 messages swept and asserts it rather than + passing vacuously. Coverage is probabilistic either way; the deterministic + guarantee is the runtime tripwire in `traversal._close`, which raises + `AlbumSplitError` when a `grouped_id` reopens after its run closed. Tested for + both `A B A` and `A B C A`. + +## Cleanup owed after running the spike + +Deleting `spike/probe.session` does **not** revoke server-side authorization. +Terminate the probe session in Telegram → Settings → Devices. This is phase 7 +step 13 and the most-missed item, because the local file looks like the whole +credential. + +## Unresolved + +1. P2, P3, P5 and P6 need a live run — no Telegram credentials exist in this + environment. +2. If P3 finds a media kind whose `.size` is absent *and* commonly large, revisit + only the estimate's `unknown size` line; no code change is implied.