mirror of
https://github.com/tiennm99/telegram-exporter.git
synced 2026-10-14 04:16:31 +00:00
docs: sync the plan with implementation and review outcomes
Phases 2-6 complete, phase 1 partial, phase 7 pending. Records what was settled without live credentials: offset_id is an exclusive lower bound under reverse=True, read from Telethon's iterator rather than inferred from one sample, so the cursor stores the last handled id. Also logs the defects found along the way and the two critical findings from the post-implementation review, plus the one finding left unresolved rather than decided unilaterally - whether a file that exhausted its retries should stay abandoned behind a set completed_at.
This commit is contained in:
tiennm99 committed
1 parent
30f68b43ff
commit
a12ddcb9c0
10 files changed
+1910
No files matched your search
@@ -0,0 +1,139 @@
|
||||
---
|
||||
phase: 1
|
||||
title: "Verification Spike"
|
||||
status: partial
|
||||
priority: P1
|
||||
dependencies: []
|
||||
effort: "~45 min"
|
||||
---
|
||||
|
||||
# Phase 1: Verification Spike
|
||||
|
||||
## Overview
|
||||
|
||||
Throwaway script settling the unverified items in the research confidence ledger against the live API, before production code depends on them. The docs page truncated mid-signature; guessing means silent resume bugs discovered after a 40 GB run.
|
||||
|
||||
**Ordering is load-bearing in this phase.** `.gitignore` is committed *first*, before anything can create a credential. The spike uses its own session path so phase 2's "first production login" remains a real, verifiable event.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: answer all six probes; record findings; amend contradicted phases
|
||||
- Non-functional: throwaway code, not shipped; no credential exists before `.gitignore` is committed; one test download only
|
||||
|
||||
## Probes
|
||||
|
||||
### P1 — `offset_id` direction under `reverse=True` (highest risk)
|
||||
|
||||
Semantics invert between sweep directions. Wrong here means resume silently skips or re-downloads everything.
|
||||
|
||||
```python
|
||||
ids = [m.id async for m in client.iter_messages(entity, limit=5)] # newest-first
|
||||
mid = sorted(ids)[2]
|
||||
fwd = [m.id async for m in client.iter_messages(entity, reverse=True, offset_id=mid, limit=3)]
|
||||
assert all(i > mid for i in fwd), f"offset_id not exclusive-lower-bound under reverse: {fwd} vs {mid}"
|
||||
```
|
||||
|
||||
Record whether `mid` itself is included. **Probe `min_id=mid` the same way** — it is documented as an exclusive lower bound and may be the more robust parameter regardless. The result decides whether the cursor stores "last handled id" or "next id to fetch".
|
||||
|
||||
### P2 — 2FA / login flow
|
||||
|
||||
Record whether `SessionPasswordNeededError` is raised and whether `client.start(phone=..., password=<callable>)` handles it. Confirm a second run against the same session needs no interaction.
|
||||
|
||||
2FA password is `getpass`-only — never argv, never env, never a file. (This supersedes any earlier wording permitting env.)
|
||||
|
||||
### P3 — `message.file` attribute surface
|
||||
|
||||
**Open question, not a confirmation.** For one photo, one video, one document, one voice note, one text-only message, and anything unusual in reach (poll, webpage preview, expiring media):
|
||||
|
||||
```python
|
||||
f = msg.file
|
||||
print(msg.id, msg.grouped_id, type(msg.media).__name__,
|
||||
getattr(f, 'name', None), getattr(f, 'ext', None),
|
||||
getattr(f, 'size', None), getattr(f, 'mime_type', None))
|
||||
```
|
||||
|
||||
Answer: **for which media kinds is `.size` absent?** Do not assume it is always present — phases 3, 5, and 6 each need a rule for `None`.
|
||||
|
||||
### P4 — `download_media` return value and file-object support
|
||||
|
||||
- Download one photo and one document with `file=<path>`; record the exact return type/value. Phase 5 uses this as the source of truth for `os.replace`.
|
||||
- Test whether `download_media(msg, file=<open file object>)` works across media branches. Phase 5 needs to own the fd to `fsync` it before rename. If it does not work, the fallback is download-to-path then re-open and fsync.
|
||||
|
||||
### P5 — photo `.size` vs bytes on disk (NEW — ledger item the original probe set missed)
|
||||
|
||||
```python
|
||||
p = await client.download_media(photo_msg, file=str(tmp))
|
||||
print("declared", photo_msg.file.size, "actual", os.path.getsize(p),
|
||||
"delta", os.path.getsize(p) - photo_msg.file.size)
|
||||
```
|
||||
|
||||
Phase 6 documents photo sizes as approximate while phase 5 originally used exact equality as a hard gate — that contradiction was the poison pill this probe closes. The resolved design is answer-independent; the answer only improves report labelling and validator tuning.
|
||||
|
||||
### P6 — album adjacency across a pagination boundary
|
||||
|
||||
The original probe capped `limit ≤ 5`, which cannot cross Telethon's ~100-message page boundary and so could not test the thing it existed to test. **Cap downloads at 1; do not cap history reads.**
|
||||
|
||||
```python
|
||||
rows = [(m.id, m.grouped_id) async for m in
|
||||
client.iter_messages(entity, reverse=True, offset_id=start, limit=350)]
|
||||
assert len(rows) >= 250, "window too small to cross a page boundary — probe is vacuous"
|
||||
ids = [i for i, _ in rows]
|
||||
assert ids == sorted(ids), "sweep is not strictly ascending"
|
||||
runs = [k for k, _ in itertools.groupby(rows, key=lambda r: r[1])]
|
||||
gids = [g for g in runs if g is not None]
|
||||
assert len(gids) == len(set(gids)), f"a grouped_id occupies >1 run — albums DO split: {gids}"
|
||||
```
|
||||
|
||||
Coverage is probabilistic (3+ pages, several albums). The deterministic guarantee is phase 3's runtime tripwire; this probe is the cheap prior.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `.gitignore` (final version, committed first), `requirements.txt`
|
||||
- Create: `spike/probe_telethon_api.py` (throwaway, gitignored)
|
||||
- Create: `plans/reports/spike-260811-1411-telethon-api-verification-report.md`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Write the **final** `.gitignore` and commit it. Nothing else exists yet:
|
||||
```
|
||||
.venv/
|
||||
*.session
|
||||
*.session-journal
|
||||
*.session-wal
|
||||
*.session-shm
|
||||
spike/
|
||||
exports/
|
||||
.export-state.json
|
||||
.export.lock
|
||||
__pycache__/
|
||||
*.egg-info/
|
||||
.env
|
||||
```
|
||||
2. **Gate — must pass before step 3:** `git check-ignore -q .venv spike/ && echo OK` (exit 0)
|
||||
3. `python3 -m venv .venv && .venv/bin/pip install "telethon>=1.41,<2"`, then `.venv/bin/pip freeze > requirements.txt` — this records the exact version the probes verified
|
||||
4. `umask 077; mkdir spike`
|
||||
5. Obtain `api_id`/`api_hash` from my.telegram.org; export as `TG_API_ID`/`TG_API_HASH` (never hardcode)
|
||||
6. Write and run `spike/probe_telethon_api.py` (P1–P6), session at `spike/probe.session`
|
||||
7. Write the findings report, marking each ledger item CONFIRMED or CORRECTED, and recording `telethon.__version__`
|
||||
8. **If any finding contradicts the plan, amend the affected phase file before proceeding**
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `.gitignore` is committed before any venv, session, or credential exists — verified by `git log`
|
||||
- [ ] `git check-ignore -q spike/ .venv` exits 0 before step 3 runs
|
||||
- [ ] P1: `offset_id` inclusivity/direction under `reverse=True` recorded, and `min_id` compared
|
||||
- [ ] P2: 2FA flow documented; second run confirmed non-interactive
|
||||
- [ ] P3: `.size` availability tabulated per media kind, including any kind where it is absent
|
||||
- [ ] P4: `download_media` return value recorded; file-object support determined
|
||||
- [ ] P5: photo declared-vs-actual size delta recorded
|
||||
- [ ] P6: ≥250 messages swept, ascending order asserted, no `grouped_id` occupying two runs
|
||||
- [ ] `requirements.txt` records the verified Telethon version
|
||||
- [ ] Findings report written; contradicted phases amended
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Wrong `offset_id` assumption reaches production** → the point of this phase; assertions must run, not be eyeballed.
|
||||
- **Credential created before `.gitignore`** → step 1 precedes step 3, with an explicit gate between them.
|
||||
- **Probe capped too small to test what it claims** → P6 requires ≥250 messages and asserts it. History reads are cheap (~4 requests); only *downloads* are meaningfully rate-limited.
|
||||
- **Spike session left live** → deleting `spike/probe.session` does **not** revoke server-side authorization. Phase 7 cleanup must terminate it in Telegram → Settings → Devices.
|
||||
- **Credentials leaked into the spike script** → env vars only; `spike/` gitignored from commit one.
|
||||
+215
@@ -0,0 +1,215 @@
|
||||
---
|
||||
phase: 2
|
||||
title: "Skeleton and Session Auth"
|
||||
status: complete
|
||||
priority: P1
|
||||
dependencies: [1]
|
||||
effort: "~2h"
|
||||
---
|
||||
|
||||
# Phase 2: Skeleton and Session Auth
|
||||
|
||||
## Overview
|
||||
|
||||
Package skeleton, the authenticated Telethon client, the flood-wait primitives every network call routes through, and the process-level safety rails: umask, logging policy, git-safety assertion, and the single-instance lock.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: one-time interactive login; later runs non-interactive; `tg-export --help` after `pip install -e .`
|
||||
- Non-functional: no file is ever group/world-readable at any instant; no credential in argv, env-where-forbidden, or logs; two concurrent runs cannot corrupt each other
|
||||
|
||||
## Architecture
|
||||
|
||||
### Process rails — `cli.main()`, before anything else
|
||||
|
||||
```python
|
||||
os.umask(0o077) # FIRST statement. Covers .session and every sqlite sibling,
|
||||
# export tree, messages.jsonl, .part, state, lock — one line, no per-file chmod.
|
||||
```
|
||||
|
||||
Verified empirically: `sqlite3.connect()` on a fresh path yields `0644` under a default umask, and the auth key is written *during* login — so a post-hoc `chmod` is provably too late. With `umask(0o077)` set first, the db and its `-journal` are both `0600`.
|
||||
|
||||
### Session path
|
||||
|
||||
```python
|
||||
SUFFIX = ".session"
|
||||
|
||||
def prepare_session_path(p: Path) -> Path:
|
||||
# Telethon appends '.session' when absent — chmod-ing the un-suffixed path protects nothing
|
||||
p = p if p.name.endswith(SUFFIX) else p.with_name(p.name + SUFFIX)
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
try:
|
||||
os.close(os.open(p, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600))
|
||||
except FileExistsError:
|
||||
os.chmod(p, 0o600)
|
||||
return p
|
||||
|
||||
def assert_session_private(p: Path) -> None: # this is the test, not a checkbox
|
||||
for f in p.parent.glob(p.name + "*"):
|
||||
if stat.S_IMODE(f.stat().st_mode) & 0o077:
|
||||
raise SystemExit(f"insecure mode on {f}: {oct(f.stat().st_mode)}")
|
||||
```
|
||||
|
||||
Default path `${XDG_STATE_HOME:-~/.local/state}/tg-export/<name>.session` — **outside the repo**, so the safe location is the default and the git check only bites deliberate choices.
|
||||
|
||||
### Git safety — robust to `--out` / `--session` moving the targets
|
||||
|
||||
Adding `.gitignore` patterns cannot help when the operator can point a flag anywhere. Assert at runtime:
|
||||
|
||||
```python
|
||||
def assert_not_committable(p: Path) -> None:
|
||||
"""No-op outside a git work tree. rc 0=ignored, 1=NOT ignored, 128=not a repo."""
|
||||
try:
|
||||
rc = subprocess.run(["git", "check-ignore", "-q", str(p)],
|
||||
cwd=p.parent, capture_output=True).returncode
|
||||
except FileNotFoundError:
|
||||
return
|
||||
if rc == 1:
|
||||
raise SystemExit(f"refusing: {p} is inside a git repo and is not gitignored.\n"
|
||||
f"add it to .gitignore, or choose a path outside the repo.")
|
||||
```
|
||||
|
||||
Called on the session path and export root at startup. Exit codes verified empirically, including on nonexistent paths.
|
||||
|
||||
### Single-instance lock
|
||||
|
||||
```python
|
||||
@contextmanager
|
||||
def exclusive(path: Path):
|
||||
"""flock releases on process death, kill -9 included — no stale-lock cleanup to get wrong."""
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
f = open(path, "a+")
|
||||
try:
|
||||
fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
f.seek(0)
|
||||
raise SystemExit(f"busy: another tg-export holds {path} [{f.read().strip() or '?'}]")
|
||||
f.seek(0); f.truncate()
|
||||
f.write(f"pid={os.getpid()} host={socket.gethostname()} started={now_iso()}\n"); f.flush()
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
f.close()
|
||||
```
|
||||
|
||||
Two call sites: `<export_root>/.export.lock` (covers `.part` files, state, sidecar) and `<session>.lock` (two Telethon clients on one SQLite session is what Telethon itself warns about). Per-root, not global — exporting two different groups concurrently is legitimate.
|
||||
|
||||
Contention: fail fast, exit 3, no wait, no `--force`. The message names pid/host/start-time, so "did it hang?" gets an answer instead of a second process. **Dry-run takes no lock** — it writes nothing.
|
||||
|
||||
### Logging policy
|
||||
|
||||
```python
|
||||
def configure_logging(verbose: bool) -> None:
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||
logging.getLogger("telegram_exporter").setLevel(logging.DEBUG if verbose else logging.INFO)
|
||||
logging.getLogger("telethon").setLevel(logging.WARNING) # unconditional floor
|
||||
logging.getLogger("asyncio").setLevel(logging.WARNING)
|
||||
```
|
||||
|
||||
Never touch the root level. `--verbose` means *our* logger only — telethon's request logging carries `api_id`, `phone_number`, and `phone_code_hash`. No `--debug-telethon` flag.
|
||||
|
||||
Note: CPython does not print local variables in tracebacks, so no exception-scrubbing machinery is needed. Just don't install `rich` or `better-exceptions`.
|
||||
|
||||
### Credential policy — one table, referenced by every other doc
|
||||
|
||||
| Credential | Source | Why |
|
||||
|---|---|---|
|
||||
| `api_id`, `api_hash` | env `TG_API_ID` / `TG_API_HASH` only | needed every run; revocable at my.telegram.org |
|
||||
| phone | prompt; `TG_PHONE` optional | PII, not a secret |
|
||||
| login code | **prompt only** | single-use, 5-minute TTL |
|
||||
| 2FA password | **`getpass` only** — no env, no file, no flag | once per session lifetime; prompting costs nothing |
|
||||
|
||||
`.env` **may** contain `TG_API_ID`, `TG_API_HASH`, `TG_PHONE`. It **must not** contain the 2FA password, login code, or session data. The app does **not** read `.env` — no `python-dotenv` to read a file the shell already handles (`set -a; . ./.env; set +a`).
|
||||
|
||||
### Flood-wait primitives — two, not one
|
||||
|
||||
The single-value wrapper cannot protect the sweep: `iter_messages` is consumed with `async for`, and Telethon raises `FloodWaitError` on the *next page fetch*, deep inside the loop — so it escapes a wrapper that only guards construction, and re-invoking the factory would restart the sweep from `after_id`.
|
||||
|
||||
```python
|
||||
def _flood_sleep(e, max_wait_s, where=""):
|
||||
"""max_wait_s is None by default -> sleep however long Telegram demands.
|
||||
Always log the WAKE TIME, not just the duration: an unattended 4-hour sleep must
|
||||
read as a sleep, not as a hang. This is the mitigation for the no-ceiling default."""
|
||||
if max_wait_s is not None and e.seconds > max_wait_s:
|
||||
raise Abort(f"flood wait {e.seconds}s exceeds --max-flood-wait {max_wait_s}s", 6)
|
||||
wake = datetime.now(UTC) + timedelta(seconds=e.seconds)
|
||||
log.warning("flood wait %ss%s — sleeping until %s", e.seconds, where, wake.isoformat())
|
||||
return asyncio.sleep(e.seconds + random.uniform(1.0, 3.0))
|
||||
|
||||
async def with_flood_retry(make_coro, *, max_wait_s: int | None = None):
|
||||
"""Download path. Never tight-retry — retries during an active wait are fresh violations."""
|
||||
while True:
|
||||
try:
|
||||
return await make_coro()
|
||||
except FloodWaitError as e:
|
||||
await _flood_sleep(e, max_wait_s)
|
||||
|
||||
async def aiter_with_flood_retry(make_agen, *, start_id, max_wait_s: int | None = None):
|
||||
"""Sweep path. Catches mid-iteration waits and rebuilds the generator from the
|
||||
last id actually yielded, so no message is skipped or replayed."""
|
||||
last = start_id
|
||||
while True:
|
||||
try:
|
||||
async for item in make_agen(last):
|
||||
last = item.id
|
||||
yield item
|
||||
return
|
||||
except FloodWaitError as e:
|
||||
await _flood_sleep(e, max_wait_s, where=f" mid-sweep after id {last}")
|
||||
```
|
||||
|
||||
`max_wait_s=None` is the default per the validation decision: unattended completion is preferred over failing fast. The wake-time log is what keeps that choice diagnosable.
|
||||
|
||||
Also set `client.flood_sleep_threshold = 120` so routine short waits are absorbed inside Telethon.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `pyproject.toml`, `README.md`
|
||||
- Create: `src/telegram_exporter/{__init__,cli,session}.py`
|
||||
- Create: `tests/{__init__,test_logging,test_lock}.py`
|
||||
- Modify: `.gitignore` (verify coverage; phase 1 created it)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. `pyproject.toml` — `telethon>=1.41,<2`, `[project.scripts] tg-export = "telegram_exporter.cli:main"`, hatchling backend, `dev = ["pytest>=8"]`. Install with `pip install -r requirements.txt -e .` so the phase-1-verified version is what's used.
|
||||
2. **Verify** (do not recreate) that `.gitignore` covers the production session path and export root.
|
||||
3. `cli.py`: `os.umask(0o077)` as the first statement, then `configure_logging`, argparse, `assert_not_committable`, locks, dispatch.
|
||||
4. `session.py`: credential loading with an actionable error pointing at my.telegram.org; `prepare_session_path`; client construction; login (phone/code prompts, 2FA via `getpass`); `assert_session_private` after connect; async context manager guaranteeing `disconnect()` — SQLite state is not flushed without it.
|
||||
5. `session.py`: both flood primitives and the `Abort` exception type.
|
||||
6. Entity resolution for `--group`: numeric id, `@username`, `t.me` link. Use `telethon.utils.get_peer_id(entity)` as the canonical int for the export root. Clear message when the account is not a member.
|
||||
7. Full flag surface (later phases wire their own):
|
||||
```
|
||||
tg-export --group <id|@username|link> --out DIR
|
||||
[--dry-run] [--since YYYY-MM-DD] [--until YYYY-MM-DD]
|
||||
[--types photo,video,document,audio,voice] [--max-size 100MB]
|
||||
[--include-text] [--limit N] [--session PATH] [--reset-state]
|
||||
[--min-free 2GiB] [--max-flood-wait SECONDS] [--verbose]
|
||||
```
|
||||
`--max-flood-wait` is **unset by default — the tool sleeps through any flood wait, however long** (validation decision). Passing a number imposes a ceiling above which the run exits cleanly with a resumable cursor (exit 6).
|
||||
|
||||
<!-- Updated: Validation Session 1 - added --include-text; --max-flood-wait default inverted to no ceiling -->
|
||||
|
||||
8. Tests: `test_logging.py` (telethon logger stays ≥ WARNING even with `verbose=True`), `test_lock.py` (subprocess holds the lock; second acquire exits 3).
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `pip install -r requirements.txt -e ".[dev]"` succeeds on Python 3.12 / aarch64
|
||||
- [ ] `tg-export --help` prints the full flag surface
|
||||
- [ ] Production session path does not exist before first login (asserted) — phase 1's spike session is separate
|
||||
- [ ] After login: session and every sibling matching `session*` are mode `0600`
|
||||
- [ ] Second run is non-interactive: `tg-export --group G --limit 0 </dev/null` exits 0 (EOF makes any prompt fail loudly)
|
||||
- [ ] `test_logging.py`: `configure_logging(verbose=True)` leaves `telethon` at ≥ `WARNING`
|
||||
- [ ] `test_lock.py`: concurrent acquire exits 3 without touching any file
|
||||
- [ ] `assert_not_committable` refuses a session or export path that is inside the repo and not ignored
|
||||
- [ ] `disconnect()` runs on both success and exception paths
|
||||
- [ ] 2FA password is reachable only via `getpass` — no flag, no env read, anywhere in the source
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Session file group/world-readable at any instant** → `umask(0o077)` first, `O_EXCL` pre-create, post-connect assertion. Verified: chmod-after-connect is too late *and* targets the wrong filename when `--session foo` is passed.
|
||||
- **Session committed** → gitignore from phase 1, default path outside the repo, runtime `git check-ignore` assert.
|
||||
- **Credentials in logs** → telethon logger pinned at WARNING unconditionally; enforced by test, not by checkbox.
|
||||
- **Two policies for the 2FA password** → one table here; phase 1 amended to match.
|
||||
- **Concurrent runs corrupting state** → flock on export root and session, acquired *before* the destructive `.part` sweep.
|
||||
- **Missing `disconnect()`** → context manager, not discipline.
|
||||
- **Flood wait during the sweep** → `aiter_with_flood_retry`; the single-value wrapper alone cannot cover `async for`.
|
||||
+145
@@ -0,0 +1,145 @@
|
||||
---
|
||||
phase: 3
|
||||
title: "Traversal Grouping and Filters"
|
||||
status: complete
|
||||
priority: P1
|
||||
dependencies: [2]
|
||||
effort: "~2h"
|
||||
---
|
||||
|
||||
# Phase 3: Traversal Grouping and Filters
|
||||
|
||||
## Overview
|
||||
|
||||
The single traversal generator both run modes consume: chronological sweep, album buffering, filter application. Yields `Post` objects — one per logical post a human would recognize.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: sweep entire history oldest→newest; group album members; apply filters; support resume from a cursor; never stall the cursor across filtered regions
|
||||
- Non-functional: bounded memory; no downloads; an album split converts to a loud failure, never silent corruption
|
||||
|
||||
## Architecture
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class Post:
|
||||
post_id: int # lowest message id in the FULL group (pre-filter)
|
||||
messages: list[Message] # filter-surviving members, ascending by id
|
||||
max_message_id: int # highest id in the FULL group — the cursor value
|
||||
```
|
||||
|
||||
`grouped_id` is deliberately **not** a field — the sidecar reads it off the message, and folder identity uses `post_id`. Nothing else consumed it.
|
||||
|
||||
**Invariant 1 — identity is filter-invariant.** `post_id` and `max_message_id` derive from the complete `grouped_id` group before filters drop members. Phase 4's filenames key on `message_id` for the same reason, so the file index no longer depends on the filter set either.
|
||||
|
||||
### `iter_posts` — always yields, even when empty
|
||||
|
||||
```python
|
||||
async def iter_posts(client, entity, *, after_id=0, filters, limit=None):
|
||||
buf, closed, prev = [], set(), 0
|
||||
agen = lambda since: client.iter_messages(entity, reverse=True, offset_id=since)
|
||||
async for msg in aiter_with_flood_retry(agen, start_id=after_id):
|
||||
assert msg.id > prev, f"sweep not strictly ascending: {msg.id} after {prev}"
|
||||
prev = msg.id
|
||||
if buf and (msg.grouped_id is None or msg.grouped_id != buf[0].grouped_id):
|
||||
yield _close(buf, closed, filters); buf = []
|
||||
if msg.grouped_id is None:
|
||||
yield _close([msg], closed, filters)
|
||||
else:
|
||||
buf.append(msg)
|
||||
if buf:
|
||||
yield _close(buf, closed, filters)
|
||||
|
||||
def _close(buf, closed, filters) -> Post:
|
||||
gid = buf[0].grouped_id
|
||||
if gid is not None:
|
||||
if gid in closed: # album-split tripwire
|
||||
raise AlbumSplitError(
|
||||
f"grouped_id {gid} reopened at msg {buf[0].id} after being closed — "
|
||||
f"album split across a non-adjacent boundary. Do not trust this export.")
|
||||
closed.add(gid)
|
||||
return Post(post_id=buf[0].id,
|
||||
messages=[m for m in buf if _keep(m, filters)],
|
||||
max_message_id=buf[-1].id)
|
||||
```
|
||||
|
||||
**Empty posts are yielded, not skipped.** This is the resolved answer to a decision the plan previously deferred. Skipping them stalls the cursor: a `--types video` run over a photo-heavy tail would advance `cursor_id` only at videos, so every resume re-sweeps tens of thousands of messages to download nothing. The "phantom post count" objection is a *counting* bug, fixed with one line in each consumer (`if not post.messages: continue` before counting), not a design objection.
|
||||
|
||||
**Album-split tripwire.** Consecutiveness of `grouped_id` members in a sweep is assumed by the whole design and was never verified by any source. The tripwire converts a silent split — two folders for one album, no exception, no size mismatch, dedupe reporting complete — into a loud failure. Memory is one int per album (~6 MB at 100k albums); the "flat memory" criterion is restated honestly as O(albums) rather than pretending it is O(1).
|
||||
|
||||
Tracking only `last_flushed_gid` was rejected: it catches `A B A` but misses `A B C A`, saving 6 MB and losing most of the coverage.
|
||||
|
||||
### `media_size` — one rule, three consumers (DRY)
|
||||
|
||||
```python
|
||||
def media_size(msg) -> int | None:
|
||||
"""Single source of truth. Phase 1 P3 determines which kinds return None."""
|
||||
```
|
||||
|
||||
| Consumer | Rule for `None` |
|
||||
|---|---|
|
||||
| `--max-size` filter | `size is None or size <= limit` → **include**, WARN once. Silent drops violate the no-silent-gaps ethos; unknown-size media is pathological and near-always small. |
|
||||
| download validator | accept (nothing to check) |
|
||||
| estimator | **not** `or 0` — counted in a separate `unknown size: N files (not counted)` line. An unbounded estimate error must be visible. |
|
||||
| disk guard | treated as 0, absorbed by the reserve |
|
||||
|
||||
Comparing `None <= limit` unguarded raises `TypeError` inside the generator and kills a 20-hour export — that is what this helper exists to prevent.
|
||||
|
||||
### Filters
|
||||
|
||||
| Flag | Predicate |
|
||||
|---|---|
|
||||
| `--types` | media kind ∈ set; kind from `msg.media` type + document attributes |
|
||||
| `--since` / `--until` | `msg.date` in range, UTC-aware on both sides |
|
||||
| `--max-size` | via `media_size` per the table above |
|
||||
| `--include-text` | when set, keep text-only messages too (sidecar context); default off |
|
||||
|
||||
<!-- Updated: Validation Session 1 - added --include-text -->
|
||||
|
||||
**`--include-text`** (validation decision). Default off: messages with no media are dropped. When set, text-only messages survive `_keep` and reach the sidecar with an empty `files[]`, so captions and replies have surrounding conversation. They never create a post folder and never reach `download_one`.
|
||||
|
||||
It **is** part of the stored filter set — it changes what a post contains, so running without it and then with it against the same cursor would silently skip every text record.
|
||||
|
||||
`--limit N` counts **posts with files** (counting all posts would let it terminate instantly inside a filtered region, and would now also be skewed by text-only posts) and is *not* a filter — it changes when we stop, not what a post contains, so it stays out of the state file's filter set.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `src/telegram_exporter/traversal.py`
|
||||
- Create: `tests/test_traversal.py`
|
||||
- Modify: `src/telegram_exporter/cli.py` (wire `--types`, `--since`, `--until`, `--max-size`, `--limit`)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Define `Post`, `Filters`, `AlbumSplitError`; parse CLI values into `Filters` (size-string and ISO-date parsers)
|
||||
2. Implement media-kind classification and `media_size`
|
||||
3. Implement `iter_posts` per the sketch, honoring phase 1 P1's confirmed `offset_id`/`min_id` semantics
|
||||
4. Consume the sweep through `aiter_with_flood_retry` — not the single-value wrapper
|
||||
5. Unit test with **synthetic message stubs** (no network): objects with `id`, `grouped_id`, `date`, `file`
|
||||
6. Test the tripwire explicitly: a synthetic stream where a `grouped_id` reappears must raise
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] A 3-photo album yields exactly one `Post` with 3 messages
|
||||
- [ ] Two adjacent albums yield two distinct `Post`s, not one merged
|
||||
- [ ] A standalone message between two albums yields its own `Post`
|
||||
- [ ] An album at the very end of history is flushed
|
||||
- [ ] `post_id` and `max_message_id` are unchanged when a filter removes some members — asserted
|
||||
- [ ] A fully-filtered group still yields a `Post` (with `messages == []`) so the cursor advances
|
||||
- [ ] Text-only messages contribute no files, and are dropped entirely unless `--include-text` is set
|
||||
- [ ] With `--include-text`, a text-only message survives into `Post.messages` but creates no folder and no download
|
||||
- [ ] `--limit N` counts posts with files, unaffected by `--include-text`
|
||||
- [ ] A synthetic stream reopening a closed `grouped_id` raises `AlbumSplitError`
|
||||
- [ ] A non-ascending synthetic stream trips the assertion
|
||||
- [ ] `media_size` returning `None` does not raise in any of the three consumers
|
||||
- [ ] Filters compose: `--types video --since 2026-01-01 --max-size 50MB`
|
||||
- [ ] `--limit 5` stops after 5 posts *with files*
|
||||
- [ ] Album buffer never exceeds 20 entries (asserted in code)
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Identity computed post-filter** → duplicate folders and re-downloads across filter sets. Invariant 1, plus phase 4's message-id filenames; both explicitly tested.
|
||||
- **Album split across a pagination boundary** → silent two-folder corruption. Phase 1 P6 is the prior; the tripwire is the guarantee. Phase 3's earlier dismissal of this risk was an unsourced assertion guarded by a test that structurally could not fail.
|
||||
- **Cursor stalls across filtered regions** → empty posts are yielded. Decided here, not deferred to implementation.
|
||||
- **`None` size crashes the sweep** → `media_size` plus the stated rule per consumer.
|
||||
- **Flood wait mid-sweep** → `aiter_with_flood_retry` resumes from the last yielded id.
|
||||
- **Timezone-naive comparison** → `msg.date` is UTC-aware; parse `--since`/`--until` as UTC-aware or the comparison raises.
|
||||
@@ -0,0 +1,145 @@
|
||||
---
|
||||
phase: 4
|
||||
title: "Naming and Path Safety"
|
||||
status: complete
|
||||
priority: P1
|
||||
dependencies: [2]
|
||||
effort: "~1h"
|
||||
---
|
||||
|
||||
# Phase 4: Naming and Path Safety
|
||||
|
||||
## Overview
|
||||
|
||||
`paths.py` — **all** path construction, from the export root down to the filename leaf. Pure functions, no network, fully unit-testable, which is why it is its own module rather than a helper inside `downloader.py`.
|
||||
|
||||
Renamed from `naming.py`: the module owns directory construction too. That rename is what turns the group-title fix from a patch into a structural property.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: derive the export root, post dir, and filename; sanitize; guarantee uniqueness within a post folder
|
||||
- Non-functional: pure; deterministic across runs *and across filter sets* (dedupe depends on it)
|
||||
|
||||
## The governing invariant
|
||||
|
||||
> **Exactly one untrusted string ever becomes a path component: the filename leaf.**
|
||||
|
||||
Every other component is an `int`, enforced by a type check. This makes `safe_join`'s "post_dir is trusted" precondition true *by type* rather than by assumption.
|
||||
|
||||
Two untrusted strings were previously in play:
|
||||
|
||||
1. `DocumentAttributeFilename` — attacker-controlled, supplied by arbitrary group members. `../../../.ssh/authorized_keys` is a cheap, real attack. Always was in scope.
|
||||
2. **The group title** — server-supplied and editable by any group admin, used to build `exports/<group-slug>/`. This was *not* in scope, and `safe_join` could never have caught it because the escape happens in the parent component, before the join.
|
||||
|
||||
## Architecture
|
||||
|
||||
```python
|
||||
def export_root(out_dir: Path, peer_id: int) -> Path:
|
||||
"""peer_id from telethon.utils.get_peer_id(entity). int -> traversal impossible."""
|
||||
if not isinstance(peer_id, int):
|
||||
raise TypeError(peer_id)
|
||||
return (out_dir / f"g{peer_id}").resolve() # g-1001234567890
|
||||
|
||||
def post_dir(root: Path, post_id: int) -> Path:
|
||||
if not isinstance(post_id, int):
|
||||
raise TypeError(post_id)
|
||||
return root / str(post_id)
|
||||
|
||||
def derive_filename(msg) -> str: # note: no index parameter
|
||||
return f"{msg.id}_{sanitize(stem_for(msg), fallback=kind(msg))}{ext_for(msg)}"
|
||||
|
||||
def sanitize(name: str, *, fallback: str, max_bytes: int = 200) -> str:
|
||||
"""Reduce an untrusted string to a safe single path component."""
|
||||
|
||||
def safe_join(post_dir: Path, filename: str) -> Path:
|
||||
"""Join and assert containment. Raises on escape — never silently repairs."""
|
||||
```
|
||||
|
||||
### Why the directory is the chat id, not the title
|
||||
|
||||
The `g` prefix avoids a leading `-` (a CLI foot-gun, not a security issue). The human-readable name is written once to `<root>/title.txt` and printed in the run header — stored as data, never as a path.
|
||||
|
||||
Rejected alternatives: a sanitized title slug needs a second sanitizer with different rules for directories, a fallback for the empty case, and a collision story — more code and more risk. A `<slug>-<chat_id>` hybrid is readable but strands the state file and the downloaded tree in the old directory when the group is renamed mid-export.
|
||||
|
||||
One move fixes traversal, empty-slug collapse into the parent, cross-group collision, *and* rename-orphaning — and it is a net code deletion. The cost is that `ls exports/` shows numbers.
|
||||
|
||||
### Why filenames key on `message_id`, not a positional index
|
||||
|
||||
One Telegram message carries at most one media, so `message_id` cannot collide inside a post folder. Uniqueness becomes structural rather than positional.
|
||||
|
||||
A positional index — even one computed pre-filter — is unstable under *history mutation*: if a member message is deleted between runs the group shrinks and every later index shifts, silently orphaning already-downloaded files. It would also force `Post` to carry the full pre-filter member list purely for naming, reintroducing the coupling Invariant 1 exists to remove.
|
||||
|
||||
Accepted cost: `1042/1042_photo.jpg` is redundant for standalone posts. Worth it — files stay self-identifying when moved out of their folder, and `grep 1043 messages.jsonl` joins sidecar to disk with no index arithmetic.
|
||||
|
||||
### Sanitization rules
|
||||
|
||||
Derivation order: `msg.file.name` when present → `msg.file.ext`/mime with a kind-based stem (`photo`, `video`, `voice` carry no filename) → `.bin`.
|
||||
|
||||
- Take `Path(name).name` only — discards every directory component, including `..`
|
||||
- Strip NUL and control chars (`< 0x20`); replace path separators and reserved chars
|
||||
- Strip leading dots and dashes (hidden files, argument-looking names)
|
||||
- Collapse whitespace runs
|
||||
- Truncate to `max_bytes` counted in **UTF-8 bytes, not characters** — ext4's limit is 255 bytes and unicode names overflow well before 255 chars. Preserve the extension.
|
||||
- Empty or fully-stripped result → `fallback`
|
||||
|
||||
Final defense in `safe_join`:
|
||||
|
||||
```python
|
||||
resolved = (post_dir / filename).resolve()
|
||||
if not resolved.is_relative_to(post_dir.resolve()):
|
||||
raise ValueError(f"path escape attempt: {filename!r}")
|
||||
```
|
||||
|
||||
Sanitization should make this unreachable; reaching it means sanitization has a hole — hence raise, never repair.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `src/telegram_exporter/paths.py`
|
||||
- Create: `tests/test_paths.py`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Implement `sanitize` per the rules above
|
||||
2. Implement `derive_filename` using phase 1 P3's findings on `message.file` availability
|
||||
3. Implement `export_root` / `post_dir` with `isinstance` guards
|
||||
4. Implement `safe_join` with the containment assertion
|
||||
5. Write the table-driven tests below
|
||||
6. `pytest tests/test_paths.py` must pass before phase 5 consumes this module
|
||||
|
||||
## Test Table
|
||||
|
||||
| Input | Expected |
|
||||
|---|---|
|
||||
| `../../../etc/passwd` | `passwd` |
|
||||
| `a/b/c.jpg` | `c.jpg` |
|
||||
| `..` / `.` / `""` | fallback |
|
||||
| `\x00evil.jpg` | `evil.jpg` |
|
||||
| `....jpg` | leading dots stripped |
|
||||
| `-rf.jpg` | leading dash stripped |
|
||||
| 300-char unicode stem | ≤200 UTF-8 bytes, extension intact |
|
||||
| same msg, two different filter sets | **identical** filename |
|
||||
| two media in one post | distinct names via distinct message ids |
|
||||
| `safe_join(dir, "../x")` | raises `ValueError` |
|
||||
| `export_root(out, "..")` | raises `TypeError` (not an int) |
|
||||
| `export_root(out, "../../etc")` | raises `TypeError` |
|
||||
| `post_dir(root, "1042; rm -rf")` | raises `TypeError` |
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Every row of the test table passes
|
||||
- [ ] No sanitized name, joined to a post dir, resolves outside it
|
||||
- [ ] `export_root` and `post_dir` reject every non-`int` input
|
||||
- [ ] A group titled `..`, `a/b`, or `""` cannot influence the output path — there is no code path from title to path
|
||||
- [ ] Truncation is byte-counted; a 300-char unicode name yields a filename ext4 accepts, extension intact
|
||||
- [ ] `derive_filename` is deterministic across runs **and across filter sets**
|
||||
- [ ] Tests run offline with no Telethon client
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Path traversal via filename** → sanitize + `safe_join` assertion + table tests.
|
||||
- **Path traversal via group title** → structurally impossible: no code path exists from title to path component.
|
||||
- **Filter-dependent filenames** → keyed on `message_id`; asserted by the two-filter-sets test row.
|
||||
- **Index instability under message deletion** → no positional index exists.
|
||||
- **Char-based truncation on unicode** → `OSError: File name too long` mid-export. Byte-counted.
|
||||
- **Non-deterministic fallbacks** (e.g. a timestamp) → dedupe breaks and every re-run re-downloads. Fallbacks derive only from message id and kind.
|
||||
- **Windows reserved names** (`CON`, `PRN`) → not applicable on Linux; deliberately skipped.
|
||||
+268
@@ -0,0 +1,268 @@
|
||||
---
|
||||
phase: 5
|
||||
title: "Downloader State and Resume"
|
||||
status: complete
|
||||
priority: P1
|
||||
dependencies: [3, 4]
|
||||
effort: "~4h"
|
||||
---
|
||||
|
||||
# Phase 5: Downloader State and Resume
|
||||
|
||||
## Overview
|
||||
|
||||
The download path: fetch media sequentially, write durably, dedupe, persist the resume cursor and the `messages.jsonl` sidecar. This is where correctness under interruption is won or lost.
|
||||
|
||||
Governing principle: **the filesystem is the state; the state file is a hint; the sidecar is the log.** Everything below follows from it.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: download each `Post`'s media into its folder; skip completed files; resume after any interruption; record every file and every failure
|
||||
- Non-functional: concurrency 1; no partial file ever presented as complete; the cursor never points past an incomplete post
|
||||
|
||||
## Architecture
|
||||
|
||||
### Durability and dedupe are one decision, not two
|
||||
|
||||
Telethon has no byte-level resume — `download_media` writes from offset 0 every call. So resumability lives at *message* granularity, and completeness is proven by **arrival at the target path**:
|
||||
|
||||
```
|
||||
per file: write -> flush -> fsync(fd) -> os.replace(tmp, target)
|
||||
per post: fsync(post_dir) -> sidecar.append + fsync -> state.commit (tmp -> fsync -> replace -> fsync(root))
|
||||
```
|
||||
|
||||
With `fsync` before `os.replace`, a file at the target path can only have arrived fully downloaded. Therefore:
|
||||
|
||||
```python
|
||||
if target.exists():
|
||||
if target.stat().st_size > 0:
|
||||
return SKIPPED
|
||||
target.unlink() # zero-byte junk from ENOSPC; re-fetch
|
||||
```
|
||||
|
||||
**Existence, not size equality.** Size equality would only add detection of *external* corruption — a fabricated threat for a personal archive — and it is precisely what created the poison pill: phase 6 documents photo `.size` as the size *variant* Telethon selects, i.e. approximate, while an exact-equality gate would delete every completed photo on resume and then raise. The fsync is what licenses the simpler predicate.
|
||||
|
||||
Stray `.part` files are swept at startup, **after** the lock is acquired — the sweep is the destructive operation, so locking after it locks after the damage.
|
||||
|
||||
### Validator: advisory, never raises
|
||||
|
||||
```python
|
||||
actual, declared = tmp.stat().st_size, media_size(msg)
|
||||
if declared is None or actual == declared: accept
|
||||
elif kind(msg) == "photo": accept, log DEBUG # variant size, expected
|
||||
else: RETRY; on exhaustion -> FAILED
|
||||
```
|
||||
|
||||
Documents, video, and audio carry an exact `document.size`, so a mismatch there is real truncation. Photos are the only approximate case. This design is correct **whether or not** phase 1 P5 finds photo sizes exact.
|
||||
|
||||
### Error taxonomy — the failure paths get the same rigor as the happy path
|
||||
|
||||
Three buckets. `FloodWaitError` never appears here: it is owned by `with_flood_retry`, because retries during an active wait are counted as fresh violations.
|
||||
|
||||
```python
|
||||
TRANSIENT_ERRNO = {EAGAIN, EINTR, EIO, ETIMEDOUT, ECONNRESET}
|
||||
FATAL_ERRNO = {ENOSPC, EDQUOT, EROFS, EACCES}
|
||||
MAX_ATTEMPTS, BACKOFF = 3, lambda n: 5 * 2 ** (n - 1) # 5, 10, 20 s
|
||||
|
||||
async def download_one(client, entity, post_dir, msg) -> Result:
|
||||
target = safe_join(post_dir, derive_filename(msg))
|
||||
if target.exists():
|
||||
if target.stat().st_size > 0: return SKIPPED
|
||||
target.unlink()
|
||||
check_disk(msg) # before the first byte
|
||||
tmp, refreshed = target.with_name(target.name + ".part"), False
|
||||
|
||||
for attempt in range(1, MAX_ATTEMPTS + 1):
|
||||
try:
|
||||
with open(tmp, "wb") as f:
|
||||
await with_flood_retry(lambda: client.download_media(msg, file=f))
|
||||
f.flush(); os.fsync(f.fileno())
|
||||
if validate(tmp, msg) is RETRY:
|
||||
tmp.unlink(missing_ok=True); await sleep(BACKOFF(attempt)); continue
|
||||
os.replace(tmp, target)
|
||||
return DOWNLOADED(target)
|
||||
|
||||
# ---- SKIP: this message is unavailable; the run continues ----
|
||||
except FileReferenceExpiredError:
|
||||
tmp.unlink(missing_ok=True)
|
||||
if refreshed: return FAILED(msg, "file reference expired")
|
||||
msg = await client.get_messages(entity, ids=msg.id); refreshed = True
|
||||
if msg is None or msg.media is None: return FAILED(msg, "message deleted")
|
||||
continue
|
||||
except (MediaEmptyError, FileIdInvalidError, FilerefUpgradeNeededError):
|
||||
tmp.unlink(missing_ok=True); return FAILED(msg, "media unavailable or expired")
|
||||
|
||||
# ---- ABORT: the run cannot continue ----
|
||||
except (AuthKeyUnregisteredError, AuthKeyDuplicatedError,
|
||||
SessionRevokedError, UserDeactivatedBanError):
|
||||
tmp.unlink(missing_ok=True); raise Abort("session invalidated — re-login", 4)
|
||||
except (ChannelPrivateError, ChatForbiddenError, ChannelInvalidError):
|
||||
tmp.unlink(missing_ok=True); raise Abort("lost access to the group", 5)
|
||||
|
||||
except OSError as e: # fatal checked FIRST
|
||||
tmp.unlink(missing_ok=True)
|
||||
if e.errno in FATAL_ERRNO: raise Abort(f"filesystem: {e}", 3)
|
||||
if e.errno in TRANSIENT_ERRNO: await sleep(BACKOFF(attempt)); continue
|
||||
raise Abort(f"unexpected OS error: {e}", 1)
|
||||
|
||||
except (ConnectionError, asyncio.TimeoutError, asyncio.IncompleteReadError,
|
||||
rpcerrorlist.TimedOutError, rpcerrorlist.RpcCallFailError):
|
||||
tmp.unlink(missing_ok=True); await sleep(BACKOFF(attempt)); continue
|
||||
|
||||
return FAILED(msg, f"exhausted {MAX_ATTEMPTS} attempts")
|
||||
```
|
||||
|
||||
`FileReferenceExpiredError` handling is **not optional** — file references expire in hours, so a 20-hour run will hit it. One `get_messages` refresh is four lines and the highest-value item in this table.
|
||||
|
||||
If phase 1 P4 finds `download_media` rejects a file object, fall back to download-to-path then re-open and fsync before rename.
|
||||
|
||||
### The sink loop — Invariant 2 by construction
|
||||
|
||||
```python
|
||||
async def run_download(posts, state, sidecar, root):
|
||||
try:
|
||||
async for post in posts:
|
||||
media, results = [m for m in post.messages if m.media], [] # results must be bound
|
||||
if media: # text-only members download nothing
|
||||
results = [await download_one(...) for msg in media]
|
||||
fsync_dir(post_dir)
|
||||
if post.messages: # may be text-only under --include-text
|
||||
sidecar.append(post, results); sidecar.fsync()
|
||||
if wrote_files or (monotonic() - last_commit) > 5.0:
|
||||
state.commit(post.max_message_id) # only after a COMPLETE post
|
||||
state.mark_completed()
|
||||
except Abort as a:
|
||||
log.error(a.reason); sys.exit(a.code) # cursor already correct
|
||||
except KeyboardInterrupt:
|
||||
log.info("interrupted — re-run the same command to resume"); sys.exit(130)
|
||||
finally:
|
||||
await client.disconnect()
|
||||
```
|
||||
|
||||
**The cursor is never written from an abort path**, so it cannot point past an in-flight post. "Commit the cursor and exit" is deleted as a concept — it was the mechanism by which the disk guard could have violated Invariant 2.
|
||||
|
||||
The 5-second throttle stops 100k filtered messages from causing 100k state writes. Losing an uncommitted sweep-only advance costs a cheap re-sweep with zero downloads.
|
||||
|
||||
Exit codes: `0` ok · `1` unexpected · `2` dry-run won't fit · `3` disk or lock contention · `4` auth · `5` access · `6` flood ceiling (**only reachable when `--max-flood-wait` is explicitly set** — unset means sleep indefinitely) · `7` filter/chat mismatch · `130` interrupt.
|
||||
|
||||
### Disk guard — one mechanism, per file
|
||||
|
||||
```python
|
||||
def check_disk(msg):
|
||||
need = (media_size(msg) or 0) + cfg.min_free # default 2 GiB
|
||||
if shutil.disk_usage(out_dir).free < need:
|
||||
raise Abort(f"insufficient disk: need {need}, have {free}", 3)
|
||||
```
|
||||
|
||||
Replaces all three previous mechanisms. `shutil.disk_usage` is one `statvfs` — microseconds, unmeasurable against multi-second downloads at concurrency 1. The every-50-files poll optimized a cost that does not exist while opening a ~100 GB blind spot: a single Telegram file reaches 2–4 GB, so up to ~100 GB could be written between two polls on a 41 GB-free host. Comparing against *this file's* size closes that structurally. The 2 GiB reserve is principled — it covers one entire unknown-size file at the non-premium upload cap.
|
||||
|
||||
### State file
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"chat_id": -1001234567890,
|
||||
"chat_title": "My Group",
|
||||
"filters": {"types": ["photo","video"], "since": "2026-01-01T00:00:00Z",
|
||||
"until": null, "max_size": 104857600, "include_text": false},
|
||||
"cursor_id": 1042,
|
||||
"completed_at": null,
|
||||
"started_at": "2026-08-11T07:00:00Z",
|
||||
"updated_at": "2026-08-11T07:42:11Z"
|
||||
}
|
||||
```
|
||||
|
||||
- **One cursor field.** `swept_to_id` vs `last_done_id` is a distinction without a difference here: the sweep is a generator driven by the sink at concurrency 1, so it can never run ahead. Two fields would invent a divergence to manage. `cursor_id` means: *every post with `max_message_id <= cursor_id` has been handled under `filters`*.
|
||||
- **Filters stored verbatim; the sha256 fingerprint is deleted.** A digest is one-way, and two phases require the tool to *name* what changed. Comparing dicts does that for free, with less code than hashing.
|
||||
- **`counters` deleted.** Derivable from the sidecar and double-counting on every resume (re-processed albums re-increment), which would make phase 7 chase phantom discrepancies. Each run prints its own in-memory session totals at exit.
|
||||
- **`completed_at`** — set on clean exhaustion, cleared to `null` at every run start. The only way to answer "did my 20-hour export finish?" without guessing.
|
||||
- Mismatch on `filters` or `chat_id` → refuse, print the diff, exit 7:
|
||||
```
|
||||
filter mismatch:
|
||||
types: ['photo','video'] -> ['video']
|
||||
max_size: 100MB -> (none)
|
||||
re-run with the original filters, or --reset-state to start over
|
||||
```
|
||||
|
||||
`--limit` is **not** in the filter set — it changes when we stop, not what a post contains, so including it would make a limited test run poison every later full run. A `--limit 20` run is a resumable partial export whose cursor genuinely reflects completed work.
|
||||
|
||||
### Sidecar
|
||||
|
||||
```json
|
||||
{"message_id":1043,"post_id":1042,"grouped_id":88,"date":"2026-07-04T12:00:00Z",
|
||||
"sender_id":777,"sender_name":"Alice","caption":"beach trip","reply_to":1001,
|
||||
"files":[{"path":"1042/1043_photo.jpg","size":1234567,"declared_size":1234999,
|
||||
"name":"photo.jpg","mime":"image/jpeg","kind":"photo"}],
|
||||
"errors":[]}
|
||||
```
|
||||
|
||||
Under `--include-text` a text-only message produces the same record shape with `"files":[]` — no folder is created for it. Consumers distinguish media from context by `files` being empty, not by a separate record type.
|
||||
|
||||
<!-- Updated: Validation Session 1 - --include-text record shape; include_text joins the stored filter set -->
|
||||
|
||||
`size` is bytes on disk; `declared_size` appears only when it differs — which incidentally accumulates empirical data on photo-variant accuracy. Paths are relative to the export root. `sender_name` comes from `msg.sender` **only if already cached** — never an extra `get_entity` call, which on a 100k-message sweep would be 100k extra RPCs and a flood ban.
|
||||
|
||||
**A record is an append-only event: "the files written for this message in this run."** The correct consumer rule is therefore **union over all records for a `message_id`**, not last-write-wins. Last-write-wins was actively wrong — after a filter change it would report a narrower file list than what is on disk.
|
||||
|
||||
Startup repair: scan back from EOF for the last `\n`; if the trailing segment is non-empty and not valid JSON, truncate to that offset and warn. A host crash mid-`write()` otherwise leaves a partial line that breaks every JSONL consumer.
|
||||
|
||||
`--reset-state` **rotates** `messages.jsonl` → `messages-<utc-ts>.jsonl` and starts fresh, so the current sidecar always describes exactly one filter regime. It does **not** delete downloaded files — dedupe reclaims them; deleting 30 GB to change a filter is hostile.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `src/telegram_exporter/downloader.py`, `state.py`, `sidecar.py`
|
||||
- Create: `tests/test_download_decisions.py`, `tests/test_state.py`
|
||||
- Modify: `src/telegram_exporter/cli.py` (wire the download path, `--reset-state`, `--min-free`, `--max-flood-wait`)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. `state.py`: load/create, filter and chat_id comparison with a named diff, atomic save (tmp → fsync → replace → fsync dir), `commit`, `mark_completed`
|
||||
2. `sidecar.py`: append-only JSONL writer, relative paths, fsync, startup partial-line repair, rotation on `--reset-state`
|
||||
3. `downloader.py`: `check_disk`, `validate`, `download_one` with the full taxonomy, startup `.part` sweep (after lock acquisition)
|
||||
4. `downloader.py`: `run_download` per the sketch — sidecar before cursor, cursor unreachable from abort paths
|
||||
5. Progress output: per-file line with post id, filename, size, running total; session totals at exit including `N files failed (see messages.jsonl)`
|
||||
6. Write `<root>/title.txt` once on first run
|
||||
7. Tests per the table below
|
||||
|
||||
## Test Table — `test_download_decisions.py` (tmpdir + fake downloader, no network)
|
||||
|
||||
| Case | Expected |
|
||||
|---|---|
|
||||
| target absent | DOWNLOADED |
|
||||
| target exists, non-empty | SKIPPED, no download call |
|
||||
| target exists, zero bytes | unlinked, re-downloaded |
|
||||
| stray `.part` at startup | swept |
|
||||
| `.part` present and target valid | target wins, `.part` removed |
|
||||
| `media_size` is `None` | stated policy, asserted |
|
||||
| `file.name == "../../../etc/passwd"` | lands inside `post_dir` |
|
||||
| free space < file size + reserve | `Abort(3)` before any write |
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Full run downloads every media file into album-aware folders
|
||||
- [ ] `kill -9` mid-run then re-run: no re-downloads, no gaps, no orphan `.part`
|
||||
- [ ] Interrupting mid-album re-downloads that album and leaves no duplicates
|
||||
- [ ] Every row of the test table passes
|
||||
- [ ] Re-running with changed filters refuses and prints a per-filter diff (exit 7)
|
||||
- [ ] A different `chat_id` against an existing state file refuses (exit 7)
|
||||
- [ ] `messages.jsonl` has one line per media message with correct relative paths; a truncated final line is repaired at startup
|
||||
- [ ] `--reset-state` rotates the sidecar and preserves downloaded files
|
||||
- [ ] Media with a dead file reference is refreshed once, then recorded in `errors[]` and **skipped** — the run continues past it
|
||||
- [ ] With `--include-text`, text-only messages appear in the sidecar with `"files":[]` and create no folder
|
||||
- [ ] Toggling `--include-text` against an existing cursor triggers the filter-mismatch refusal (exit 7)
|
||||
- [ ] A flood wait logs its duration **and computed wake time**; with no `--max-flood-wait` set, the run sleeps rather than exiting
|
||||
- [ ] Simulated low disk aborts before writing, with a resumable cursor
|
||||
- [ ] `completed_at` is set only on clean exhaustion
|
||||
- [ ] `test_no_concurrency.py` (AST scan) finds no `gather`/`create_task`/`to_thread`/thread pool in `src/`
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Cursor ahead of reality** → permanent silent gaps; the worst failure here because it looks like success. Invariant 2 is structural: `state.commit` is unreachable from every abort path.
|
||||
- **Right-sized corrupt file after host crash** → fsync before `os.replace`, plus dir fsync before the cursor claims the post durable. This is what makes existence-based dedupe sound.
|
||||
- **Poison-pill validator** → validator never raises; photos excused; design independent of phase 1 P5's answer.
|
||||
- **Unhandled exception kills an unattended 20-hour run** → full taxonomy above; transient errors retry with backoff, terminal ones exit cleanly with a correct cursor.
|
||||
- **Filter/cursor confusion** → verbatim filters, named diff, hard refusal.
|
||||
- **Disk exhaustion mid-write** → per-file pre-check against this file's size plus a 2 GiB reserve.
|
||||
- **Concurrent runs** → flock acquired before the `.part` sweep (phase 2).
|
||||
- **Sidecar misread after a filter change** → union semantics, documented; `--reset-state` rotates.
|
||||
- **Temptation to parallelize when it feels slow** → flood-wait escalation to multi-hour lockouts. Enforced by an AST test, not by discipline.
|
||||
@@ -0,0 +1,102 @@
|
||||
---
|
||||
phase: 6
|
||||
title: "Dry-Run Estimator"
|
||||
status: complete
|
||||
priority: P2
|
||||
dependencies: [5]
|
||||
effort: "~1h"
|
||||
---
|
||||
|
||||
# Phase 6: Dry-Run Estimator
|
||||
|
||||
## Overview
|
||||
|
||||
`--dry-run`: sweep, download nothing, report what a real run would fetch **from where it would actually start**. With 41 GB free and an unknown group size, this is the difference between a planned export and a surprise.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: counts and bytes by media type honoring the same filters *and the same cursor*; disk comparison; non-zero exit when it won't fit
|
||||
- Non-functional: no `download_media` calls; no writes anywhere, including no creation of the export root; no lock taken; no state mutation
|
||||
|
||||
## Architecture
|
||||
|
||||
`run_estimate()` drives the **same** `iter_posts` generator with the same `Filters` as `run_download()`. That shared generator is the anti-drift property — an estimator with its own traversal would report numbers that don't match reality. There is no `Sink` Protocol: both run modes own a plain `async for`, and the duplication is two lines.
|
||||
|
||||
### The cursor fix
|
||||
|
||||
Dry-run **reads** state (never writes) and starts where the real run would:
|
||||
|
||||
```python
|
||||
state = State.load(out_dir) # read-only
|
||||
start = state.cursor_id if state and state.filters == filters else 0
|
||||
```
|
||||
|
||||
Without this, `tg-export --dry-run && tg-export` fails **closed** on every resumed run: a 38 GB group with 30 GB already downloaded and 11 GB free would re-measure the whole 38 GB, exit 2, and refuse to fetch the 8 GB that fits comfortably. The user's only escape would be to bypass the guard they were told to trust.
|
||||
|
||||
Deliberately **not** doing a "scan the export dir and subtract bytes on disk" pass: with a cursor-based sweep everything before the cursor is already excluded, and the residue is a handful of failed files and one partial post. A disk-scan reconciliation is unearned complexity.
|
||||
|
||||
### Report
|
||||
|
||||
```
|
||||
Group: My Group (id -1001234567890)
|
||||
Filters: types=video,photo since=2026-01-01 max-size=100MB
|
||||
Mode: resume from message 1042 # or "full history (no prior state)"
|
||||
|
||||
Remaining: 1,204 posts / 3,891 files
|
||||
photos 1,902 files 2.1 GiB (approximate — size variant)
|
||||
videos 1,944 files 35.4 GiB
|
||||
documents 45 files 0.7 GiB
|
||||
unknown size 3 files (not counted)
|
||||
|
||||
Total: 38.2 GiB
|
||||
Free: 11.0 GiB Reserve: 2.0 GiB
|
||||
Verdict: SHORT by 29.2 GiB exit 2
|
||||
```
|
||||
|
||||
Unknown-size files get their own line rather than being coerced to zero — an unbounded estimate error must be visible. Photo totals are labelled approximate; document/video/audio are exact.
|
||||
|
||||
Posts and files count **posts with files** — neither empty posts (fully filtered groups, which phase 3 yields to keep the cursor advancing) nor text-only posts under `--include-text` are counted, on both sides of any comparison. Text messages carry no bytes, so the estimate is unaffected by the flag.
|
||||
|
||||
<!-- Updated: Validation Session 1 - --include-text does not affect posts_with_files counting -->
|
||||
|
||||
|
||||
### The verdict is advisory; the per-file guard enforces
|
||||
|
||||
Phase 5's per-file `check_disk` is the sole enforcement mechanism. This removes the previously "optional preflight behind a flag" — a flag that appeared in no flag surface — and the double sweep it implied on huge groups.
|
||||
|
||||
Say this plainly in the README: a `SHORT` verdict does not mean the run is unsafe. It means the run will stop partway with a clean, resumable cursor.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Create: `src/telegram_exporter/estimate.py`
|
||||
- Modify: `src/telegram_exporter/cli.py` (`--dry-run` dispatches to `run_estimate`)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. `EstimateSink` accumulating posts-with-files, files, and bytes by media kind, plus an unknown-size tally
|
||||
2. Human-readable byte formatting (GiB/MiB, thousands separators)
|
||||
3. `State.load` in read-only mode; derive `start` and the `Mode:` line
|
||||
4. Free space via `shutil.disk_usage(out_dir)`; compare `total + reserve` against free
|
||||
5. Report renderer per the layout; label photo totals approximate
|
||||
6. `cli.py`: `--dry-run` → `run_estimate`, taking no lock and creating no directories
|
||||
7. Verify dry-run leaves the filesystem untouched
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] `--dry-run` completes with zero `download_media` calls and zero bytes written
|
||||
- [ ] Dry-run does not create the export root, `.export-state.json`, `messages.jsonl`, or `.export.lock`
|
||||
- [ ] Dry-run takes no lock — it can run while a real export is in progress
|
||||
- [ ] On a resumed export, `Mode:` shows the cursor and totals cover only remaining work
|
||||
- [ ] `tg-export --dry-run && tg-export` succeeds on a partially completed export that fits
|
||||
- [ ] Totals broken down by media kind; unknown-size files on their own line, never as zero
|
||||
- [ ] Exit `2` when total + reserve exceeds free space; `0` otherwise
|
||||
- [ ] Filters applied identically to a real run via the shared `iter_posts`, not by duplicated logic
|
||||
- [ ] `--dry-run --limit 20` vs `--limit 20` from the same start: post and file counts **exactly equal**, bytes within **2%** (photo approximation only)
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Estimator drifts from downloader** → the number users trust becomes a lie. Prevented structurally by the shared `iter_posts`; never fork the traversal.
|
||||
- **Dry-run mutates state** → a later real run resumes from a cursor nothing downloaded. Read-only by construction; asserted.
|
||||
- **Dry-run fails closed on resume** → fixed by reading the cursor. This was the defect that made the documented safe idiom unusable.
|
||||
- **Photo approximation misread as exact** → labelled, plus the 2% tolerance is stated rather than left to judgement.
|
||||
- **Sweeping a very large history triggers flood waits** → history reads are cheap and covered by `aiter_with_flood_retry`.
|
||||
@@ -0,0 +1,104 @@
|
||||
---
|
||||
phase: 7
|
||||
title: "End-to-End Validation"
|
||||
status: pending
|
||||
priority: P1
|
||||
dependencies: [6]
|
||||
effort: "~2h + export runtime"
|
||||
---
|
||||
|
||||
# Phase 7: End-to-End Validation
|
||||
|
||||
## Overview
|
||||
|
||||
Prove the tool against a real group, with deliberate interruption. The failure modes that matter — cursor ahead of reality, split albums, flood-wait escalation — are invisible in unit tests and surface only against live data.
|
||||
|
||||
Every step below has a checkable outcome. **No step may pass by escape clause.**
|
||||
|
||||
## Requirements
|
||||
|
||||
- Functional: full export completes, resumes correctly after kill, produces a consistent sidecar
|
||||
- Non-functional: README documents setup and semantics; no secret or third-party data is committable; the spike's server-side session is revoked
|
||||
|
||||
## Validation Script
|
||||
|
||||
1. **Dry run** — `tg-export --group <g> --out ./exports --dry-run`. Record posts, files, total bytes, verdict. Confirm no export root was created.
|
||||
|
||||
2. **Limited real run** — `--limit 20`. Inspect by hand: albums collapsed into one folder, standalone messages in their own, filenames `{message_id}_{name}`, `title.txt` written, directory named `g<chat_id>`.
|
||||
|
||||
3. **Sidecar cross-check** — scripted, not by eye:
|
||||
```bash
|
||||
python3 - <<'EOF'
|
||||
# 1. every files[].path in messages.jsonl exists on disk
|
||||
# 2. every file on disk appears in the UNION of records for its message_id
|
||||
# 3. no two post folders share a grouped_id <- catches album splitting
|
||||
EOF
|
||||
```
|
||||
|
||||
4. **Idempotency** — `--reset-state`, then re-run `--limit 20`. Expect **zero downloads, all skips**.
|
||||
*(Re-running `--limit 20` without resetting would correctly continue to posts 21–40 — the cursor advanced because those 20 posts genuinely completed. Resetting is what isolates the dedupe path, which is the thing under test.)*
|
||||
|
||||
5. **Interruption** — start a full run, `kill -9` after ~50 files. Re-run. Confirm: no surviving `.part`, no re-downloads of completed files, no gaps, the in-flight album complete.
|
||||
|
||||
6. **Network loss** — drop the interface or block Telegram with iptables mid-download for ~30s, then restore. Confirm the transient path retries with backoff and the run continues. `kill -9` alone does not exercise exception propagation or reconnect, so it cannot stand in for this.
|
||||
|
||||
7. **Filter guard** — re-run with `--types video` against the existing state. Confirm it refuses, **names the changed filters**, and exits 7. Then `--reset-state` and confirm it proceeds and rotates `messages.jsonl`.
|
||||
|
||||
8. **Concurrency guard** — start a run; while it holds the lock, start a second against the same `--out`. Confirm exit 3 with pid/host/start-time, and that no file was touched. Confirm a run against a *different* group proceeds.
|
||||
|
||||
9. **Path safety** — covered by `tests/test_paths.py` and `tests/test_download_decisions.py`, which include the hostile-filename case offline. *(The previous "otherwise rely on unit tests and note no live sample existed" escape clause is deleted — it made passing the default.)* Optionally self-send a message named `../../../etc/passwd` to a scratch group for live confirmation.
|
||||
|
||||
10. **Full export** — run to completion. Confirm `completed_at` is set. Reconcile against step 1: post and file counts **exactly equal**; bytes within **2%** (photo approximation only). Any file-count delta is a bug to investigate, never accepted — expected differences are only deleted/expired media, which appear in `errors[]` and the exit summary.
|
||||
|
||||
11. **Flood-wait behavior** — grep logs: each wait slept through once, no retry storm, no restart during an active wait, and every entry carries a computed wake time. Separately confirm `--max-flood-wait 1` exits 6 cleanly with a resumable cursor, since that path is unreachable by default.
|
||||
|
||||
11b. **`--include-text`** — run with the flag on a scratch range. Confirm text-only messages appear with `"files":[]`, create no folders, and that toggling the flag against the existing cursor triggers the exit-7 refusal.
|
||||
|
||||
12. **Memory** — `/usr/bin/time -v tg-export --dry-run` → peak RSS under 200 MB. The album tripwire set is O(albums), not O(1); the criterion states the real bound.
|
||||
|
||||
13. **Revoke the spike session** — Telegram → Settings → Devices → terminate the phase 1 probe session. **Deleting `spike/probe.session` does not revoke server-side authorization**; a live auth key otherwise survives for an account you believe is clean.
|
||||
|
||||
## Related Code Files
|
||||
|
||||
- Modify: `README.md`
|
||||
- Delete: `spike/` (or leave gitignored)
|
||||
|
||||
## README Must Document
|
||||
|
||||
- Obtaining `api_id`/`api_hash`; the credential policy table (env vs prompt vs getpass; what `.env` may contain)
|
||||
- First-run login incl. 2FA; session path and why it defaults outside the repo
|
||||
- **Why the Bot API cannot do this** — the first question any reader will have
|
||||
- Resume semantics: the cursor means "handled through here *under these filters*", and why changing filters requires `--reset-state`
|
||||
- Sidecar semantics: records are append-only events; **union over records per `message_id`**, not last-write-wins
|
||||
- Export directory is `g<chat_id>` with the name in `title.txt`, and why (path safety + surviving group renames)
|
||||
- Why downloads are sequential — flood-wait escalation, and that `--workers` deliberately does not exist
|
||||
- `--include-text`: off by default; when set, text-only messages appear in the sidecar with `"files":[]` and no folder. It is part of the stored filter set, so toggling it requires `--reset-state`
|
||||
- **Flood waits have no ceiling by default** — the tool sleeps as long as Telegram demands, logging the computed wake time each time. A long sleep is a sleep, not a hang; check the log before assuming otherwise. Pass `--max-flood-wait SECONDS` to exit resumably instead
|
||||
|
||||
<!-- Updated: Validation Session 1 - --include-text docs; flood-wait default inverted -->
|
||||
|
||||
- That the dry-run verdict is **advisory**; the per-file disk check enforces. A `SHORT` verdict means the run stops partway, resumably — not that it is unsafe
|
||||
- `cryptg` is optional and not installed by default: native code in Telethon's AES path, trusted (if adopted) because the Telethon author maintains it — not because it is "CPU-side only"
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Every step 1–13, including 11b, passes with its stated outcome — no step passes by escape clause
|
||||
- [ ] Full export completes with `completed_at` set; counts reconcile within the stated tolerance
|
||||
- [ ] Kill-and-resume produces a complete export with no gaps and no duplicates
|
||||
- [ ] Network-loss interruption recovers without operator intervention
|
||||
- [ ] Filter guard fires with a named diff and clears with `--reset-state`
|
||||
- [ ] Concurrency guard exits 3; a different group is unaffected
|
||||
- [ ] `pytest` green; `pip install -r requirements.txt -e .` from a clean venv yields a working `tg-export`
|
||||
- [ ] `git status` clean of session files, exports, state, lock, and `.env`
|
||||
- [ ] Spike session terminated server-side
|
||||
- [ ] README covers every item listed above
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Group larger than free disk** → step 1 reports it; the per-file guard stops cleanly and resumably. Scope down with `--max-size`/`--types` or export to external storage.
|
||||
- **Long export interrupted by network loss** → step 6 is the proof; step 5 alone would not be.
|
||||
- **Estimate/actual discrepancy** → expected sources are deleted/expired media (visible in `errors[]`) and photo approximation. Anything else is a filter or traversal bug — investigate.
|
||||
- **Album split** → step 3's grouped_id check plus phase 3's runtime tripwire. A split that reaches disk is silent otherwise.
|
||||
- **Account flood-limited during validation** → keep `--limit` small in steps 2–8; run step 10 once, unattended.
|
||||
- **Third-party data committed** → the sidecar contains message text and sender names. Export root is gitignored and `assert_not_committable` refuses an un-ignored path inside the repo; verify before any commit.
|
||||
- **Spike credential left live** → step 13. The most-missed cleanup item, because the local file looks like the whole credential.
|
||||
@@ -0,0 +1,328 @@
|
||||
---
|
||||
title: "Telegram Group Media Exporter"
|
||||
description: "CLI tool exporting all media from a Telegram group to local disk, grouped by message/album, with resume, filters, and dry-run size estimation."
|
||||
status: in-progress
|
||||
priority: P2
|
||||
branch: "main"
|
||||
tags: [telethon, python, cli, mtproto, export]
|
||||
blockedBy: []
|
||||
blocks: []
|
||||
created: "2026-08-11T07:22:18.583Z"
|
||||
createdBy: "ck:plan"
|
||||
source: skill
|
||||
---
|
||||
|
||||
# Telegram Group Media Exporter
|
||||
|
||||
## Overview
|
||||
|
||||
CLI tool `tg-export` — downloads all media from a Telegram group to local disk, one folder per logical post (albums collapsed), with a `messages.jsonl` sidecar preserving message context. Resumable, filterable, with a dry-run size estimate.
|
||||
|
||||
Greenfield repo. Research: [research report](../reports/research-telethon-mtproto-260811-1411-telegram-group-media-exporter-report.md). Reviewed adversarially — see [Red Team Review](#red-team-review); the design below is post-review.
|
||||
|
||||
## Locked Decisions
|
||||
|
||||
Settled before planning — do not re-litigate without new evidence.
|
||||
|
||||
| Decision | Rationale |
|
||||
|---|---|
|
||||
| Telethon + Python 3.12, MTProto **user account** | Bot API is structurally incapable: no history access, 20 MB download cap |
|
||||
| Sequential downloads, **concurrency 1** | Telethon does not parallelize transfers; parallelism accelerates FloodWait escalation (7s → 4h+) |
|
||||
| Album-aware folders, id = **lowest message id** in the group | User decision; deterministic across re-runs |
|
||||
| `SQLiteSession` file (not StringSession) | Persistent local filesystem; StringSession solves a containerization problem we don't have |
|
||||
| `argparse` (not Typer/Click) | One command, ~12 flags, no subcommands |
|
||||
| `pyproject.toml` + `[project.scripts]` | Current packaging standard |
|
||||
| Telethon pinned `>=1.41,<2` + `requirements.txt` | Telethon 2.0 is alpha with a renamed API surface; the pin file records the version phase 1 actually verified |
|
||||
|
||||
## Architecture
|
||||
|
||||
Both run modes drive the **same** `iter_posts` generator with the same `Filters`. That shared generator — not any class hierarchy — is what guarantees the dry-run estimate describes what a real run would download. A forked traversal would drift and lie.
|
||||
|
||||
```
|
||||
cli.py ──> session.py ──> traversal.py ──┬──> run_download() ──> downloader.py, state.py, sidecar.py
|
||||
argparse auth, flood sweep + │
|
||||
umask wrapper album group └──> run_estimate() ──> estimate.py
|
||||
logging + filters
|
||||
locks paths.py (shared, security-critical)
|
||||
```
|
||||
|
||||
```
|
||||
src/telegram_exporter/
|
||||
cli.py argparse, umask, logging, locks, git-safety asserts, dispatch
|
||||
session.py credentials, session path prep, client, flood retry, entity resolution
|
||||
traversal.py Post, Filters, media_size(), iter_posts, album-split tripwire
|
||||
paths.py export_root, post_dir, sanitize, derive_filename, safe_join (pure)
|
||||
downloader.py download_one, error taxonomy, .part sweep, run_download()
|
||||
state.py resume cursor only
|
||||
sidecar.py messages.jsonl append writer only
|
||||
estimate.py EstimateSink, byte formatting, run_estimate()
|
||||
tests/
|
||||
test_paths.py sanitize table, safe_join, export_root type guard
|
||||
test_download_decisions.py skip/re-download/.part logic against a tmpdir
|
||||
test_traversal.py album grouping + split tripwire fires
|
||||
test_state.py filter mismatch, atomic save
|
||||
test_lock.py subprocess contention
|
||||
test_logging.py telethon logger stays >= WARNING
|
||||
test_no_concurrency.py AST scan rejects gather/create_task
|
||||
```
|
||||
|
||||
**No `Sink` Protocol.** Two implementations with one call site and no shared behavior is indirection describing a `for` loop. `run_download()` and `run_estimate()` each own their own `async for`. The anti-drift property lives in the shared `iter_posts` — restated here so nobody re-adds the Protocol believing it was load-bearing.
|
||||
|
||||
**`state.py` and `sidecar.py` are separate modules on purpose.** The correctness-critical part is the *ordering* between them, and that ordering belongs to the caller. Two objects in two modules makes the sequence visible at one call site and makes a "helpful" `commit_and_append()` — which would destroy Invariant 2 — impossible to write without noticing.
|
||||
|
||||
Output layout:
|
||||
|
||||
```
|
||||
exports/g-1001234567890/ <- directory name is the chat id, never the group title
|
||||
title.txt <- human-readable name lives here, as data
|
||||
1042/ <- album (shared grouped_id), folder = first member id
|
||||
1042_photo.jpg <- filename keyed on message id, not a positional index
|
||||
1043_photo.jpg
|
||||
1044_video.mp4
|
||||
1047/ <- standalone message
|
||||
1047_document.pdf
|
||||
messages.jsonl
|
||||
.export-state.json
|
||||
.export.lock
|
||||
```
|
||||
|
||||
**Exactly one untrusted string ever becomes a path component: the filename leaf.** Every other component is an `int` enforced by a type check. The group title is server-supplied and editable by any admin, so it is stored as data and never as a path — which also stops a group rename from orphaning an in-progress export.
|
||||
|
||||
## Three Invariants
|
||||
|
||||
Violating any of these produces silent corruption that looks like success.
|
||||
|
||||
1. **Identity is filter-invariant.** The folder is the lowest message id in the *full* `grouped_id` group; the filename is keyed on `message_id`. Neither depends on which members a filter dropped, so `--types photo` and `--types video` address the same files.
|
||||
2. **The cursor commits only after a complete post, and never from an abort path.** `state.commit()` is unreachable from every error exit, so the cursor can never point past an in-flight post.
|
||||
3. **Filters are stored verbatim in the state file.** The cursor means "handled through here *under these filters*". A mismatch refuses the run and names what changed.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
`[x]` = verified offline by the test suite. `[~]` = implemented and unit-tested,
|
||||
but the end-to-end proof is phase 7 against a live group.
|
||||
|
||||
- [~] `tg-export --group <g> --out ./exports --dry-run` reports remaining posts/files/bytes by media type and a disk verdict, writing nothing — *"writes nothing" and the per-kind breakdown are tested; the live sweep is phase 7 step 1*
|
||||
- [~] `tg-export --group <g> --out ./exports` downloads all media into album-aware folders
|
||||
- [~] Killing the run mid-export and re-running resumes with no re-downloads and no gaps — *dedupe, cursor and `.part` sweep are tested; the `kill -9` proof is phase 7 step 5*
|
||||
- [x] `messages.jsonl` links every file to its message id, post id, date, sender, and caption
|
||||
- [x] A message whose filename is `../../../etc/passwd` writes inside the post dir, proven by test
|
||||
- [x] A group titled `..` or `a/b` cannot affect the output path, proven by test — *structurally: no code path exists from title to path component*
|
||||
- [x] A run aborts before writing the first byte of any file that would not fit, leaving a resumable cursor
|
||||
- [x] A second concurrent run against the same export root exits 3 without touching any file — *real subprocess contention*
|
||||
- [x] `FloodWaitError` is slept through once and logged with its computed wake time; no tight retry; a wait beyond an explicit `--max-flood-wait` cap exits cleanly and resumably
|
||||
- [x] Media that is deleted, expired, or unavailable is recorded and **skipped**, and the run continues
|
||||
- [x] `--verbose` leaves the `telethon` logger at `WARNING`; `api_hash` and phone absent from captured output
|
||||
- [x] Session file and every sibling are mode `0600`; export tree `0700`
|
||||
- [x] `pytest` green (180 tests); `pip install -r requirements.txt -e .` from a clean venv yields a working `tg-export`
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | Name | Status |
|
||||
|-------|------|--------|
|
||||
| 1 | [Verification Spike](./phase-01-verification-spike.md) | Partial — offline probes settled, live probes await credentials |
|
||||
| 2 | [Skeleton and Session Auth](./phase-02-skeleton-and-session-auth.md) | Complete |
|
||||
| 3 | [Traversal Grouping and Filters](./phase-03-traversal-grouping-and-filters.md) | Complete |
|
||||
| 4 | [Naming and Path Safety](./phase-04-naming-and-path-safety.md) | Complete |
|
||||
| 5 | [Downloader State and Resume](./phase-05-downloader-state-and-resume.md) | Complete |
|
||||
| 6 | [Dry-Run Estimator](./phase-06-dry-run-estimator.md) | Complete |
|
||||
| 7 | [End-to-End Validation](./phase-07-end-to-end-validation.md) | Pending — requires a live group |
|
||||
|
||||
**Dependency chain:** 1 → 2 → 3 → 4 → 5 → 6 → 7. Phase 4 is pure and offline-testable, so it can run parallel to 3. Phase 1 gates everything.
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Group scale** — unknown volume. Resolved by phase 6 dry-run.
|
||||
2. **2FA enabled?** — resolved by phase 1 P2.
|
||||
3. **Group access type** — resolved by phase 1.
|
||||
4. ~~**Text-only messages in `messages.jsonl`?**~~ **Resolved (validation):** default media-bearing only, plus an `--include-text` flag that records text messages with an empty `files[]`. The flag is part of the stored filter set, since it changes what a post contains.
|
||||
5. **Deleted/expiring media** — recorded as an `errors[]` entry on that message's sidecar record and counted at exit; the run continues.
|
||||
6. **Stray `.part` files** — swept at startup, after the lock is acquired.
|
||||
7. **Unknown-size media (`file.size is None`)** — **included** and warned, counted separately in the estimate rather than as zero. Invert if disk pressure matters more than completeness.
|
||||
8. **Export dir named `g<chat_id>`, not the group title** — chosen to make path traversal structurally impossible and to survive group renames. Human name lives in `title.txt`. Reversible if you prefer readable directory names, at the cost of both properties.
|
||||
9. **Should a FAILED file be permanently abandoned?** Raised by the
|
||||
post-implementation review, and genuinely unsettled by this plan. Today a
|
||||
failure advances the cursor and is recorded in `errors[]`, which is right for
|
||||
deleted or expired media but also covers a file that exhausted its three
|
||||
retries on transient network trouble — that one is never re-attempted, and
|
||||
`completed_at` is still set. The exit summary names the count. Options if this
|
||||
matters: re-drive the range with `--reset-state`, or add a `--retry-failed`
|
||||
pass that reads the sidecar. **Not changed unilaterally — this is a scope
|
||||
decision.**
|
||||
10. **Concurrent exports of two groups need two `--session` files**, not just two
|
||||
`--out` directories, because the session SQLite database is locked too. The
|
||||
plan claimed per-root locking made concurrent exports work; it does, but only
|
||||
with that second flag. Documented in the README and in the exit-3 message.
|
||||
|
||||
## Dependencies
|
||||
|
||||
No cross-plan dependencies.
|
||||
|
||||
External: `telethon>=1.41,<2`, pinned in `requirements.txt` to the version phase 1 verified. `cryptg` is **not** installed in v1: it is a native extension sitting in Telethon's AES path, so it sees the auth key and every plaintext byte. It ships an aarch64 wheel (verified — no build step), and the reason to trust it if adopted is that the Telethon author maintains it, not that it is "CPU-side only". Don't add a supply-chain root for a throughput problem that has not been measured.
|
||||
|
||||
## Red Team Review
|
||||
|
||||
### Session — 2026-08-11
|
||||
|
||||
**Reviewers:** 4 (Security Adversary, Failure Mode Analyst, Assumption Destroyer, Scope & Complexity Critic)
|
||||
**Findings:** 39 raw → 20 deduplicated → 15 adjudicated (13 accepted, 2 rejected, 4 corrected as overstated)
|
||||
**Severity breakdown:** 7 Critical, 8 High
|
||||
|
||||
Evidence standard adapted for a greenfield repo: reviewers cited plan-file and research-report lines, since no source code exists to grep.
|
||||
|
||||
| # | Finding | Severity | Disposition | Applied To |
|
||||
|---|---------|----------|-------------|------------|
|
||||
| 1 | Photo `.size` documented approximate but used as an exact-equality gate that raises → poison pill | Critical | Accept | Phase 1, 5 |
|
||||
| 2 | Filename index computed post-filter → breaks dedupe and determinism | Critical | Accept | Phase 3, 4, 5 |
|
||||
| 3 | Download loop has no `try`; "skip deleted media" structurally impossible | Critical | Accept | Phase 5 |
|
||||
| 4 | `<group-slug>` is attacker-controlled and unsanitized | Critical | Accept | Phase 4, plan |
|
||||
| 5 | Flood-wait wrapper cannot protect the sweep | Critical | Accept (re-reasoned) | Phase 2, 3 |
|
||||
| 6 | `--limit` advances the cursor; phase 7 idempotency step unrunnable | Critical | Accept (fix relocated) | Phase 7 |
|
||||
| 7 | Phase 1 creates `.session` before any `.gitignore`; sqlite creates it `0644` before the post-hoc `chmod` | Critical | Accept | Phase 1, 2 |
|
||||
| 8 | Disk guard fires ≤50 files late; "abort before first byte" unimplemented | High | Accept | Phase 5, 6 |
|
||||
| 9 | Empty-`Post` contract deferred to the implementer; no completion marker | High | Accept | Phase 3, 5 |
|
||||
| 10 | Dry-run ignores the cursor → `--dry-run && run` fails closed on resume | High | Accept | Phase 6 |
|
||||
| 11 | No single-instance lock | High | Accept | Phase 2, 5 |
|
||||
| 12 | `--reset-state` leaves the sidecar → mixed filter generations | High | Accept | Phase 5 |
|
||||
| 13 | `msg.file.size is None` handled three incompatible ways | High | Accept | Phase 3, 5, 6 |
|
||||
| 14 | `--verbose` unmutes telethon loggers; 2FA-env policy contradiction; gitignore coupled to default paths | High | Accept (traceback half rejected) | Phase 2 |
|
||||
| 15 | Album consecutiveness across pagination unverified and untestable as guarded | High | Accept | Phase 1, 3 |
|
||||
|
||||
**Rejected:**
|
||||
- *"Fold phase 1 into phase 2."* The ordering bug inside it is real and fixed; the merge argument adds no evidence against the gate itself, which exists to settle facts phases 3/5/6 build on. Fixed the duplication instead.
|
||||
- *"Drop probe P4 (`download_media` return value) — nothing consumes it."* True as written, but finding 1's fix makes the return value the source of truth for `os.replace`.
|
||||
|
||||
**Corrected as overstated or factually wrong** (verified empirically during resolution):
|
||||
- `cryptg` "builds from sdist on ARM64, executing at install time" — **false**. It ships a `cp312-manylinux_2_28_aarch64` wheel. The plan's *wording* was still wrong; rationale rewritten.
|
||||
- `--verbose` "leaks credentials via traceback" — **false**. CPython does not print locals in tracebacks. The logger half is real and fixed.
|
||||
- Session permission race rated Critical — realistically Medium on a single-user host. Fixed anyway: it is one `umask` line.
|
||||
- "7 modules for 600 lines" — module count is not a defect metric; cohesion is. `estimate.py` kept. The `Sink` Protocol and the `state.py` bundling were the real findings.
|
||||
|
||||
**Net effect:** the resolved design is *smaller* than the reviewed one. Deleted: the `Sink` Protocol, state `counters`, the sha256 filter fingerprint, the every-50-files disk poll, the optional-preflight flag, the positional filename index, and title-based slugging.
|
||||
|
||||
### Whole-Plan Consistency Sweep
|
||||
- Files reread: plan.md, phase-01 … phase-07
|
||||
- Decision deltas checked: 15
|
||||
- Reconciled stale references: filename scheme (ph3/4/5), state schema (ph5/6/7), module names `naming.py`→`paths.py` (plan/ph4/ph5), flag surface (ph2/3/5/6), dedupe predicate (ph5/7), export dir naming (plan/ph4/ph5/ph7), disk guard (plan/ph5/ph6/ph7)
|
||||
- Unresolved contradictions: 0
|
||||
|
||||
## Implementation Log
|
||||
|
||||
### Session 1 — 2026-08-12/13 — phases 2–6
|
||||
|
||||
Phases 2–6 implemented and green: **180 tests**, all offline against synthetic
|
||||
message stubs. `pip install -r requirements.txt -e .` from a clean venv yields a
|
||||
working `tg-export` on Telethon **1.44.0** (aarch64 wheel).
|
||||
|
||||
Phase 1 could not be run — no Telegram credentials exist in this environment —
|
||||
but most of what it gates was settled without them, recorded in the
|
||||
[spike findings report](../reports/spike-260811-1411-telethon-api-verification-report.md):
|
||||
|
||||
- **P1 CONFIRMED from Telethon's source**, not from a sample: under `reverse=True`
|
||||
the iterator does `offset_id += 1` before the first request and again per page,
|
||||
and `min_id` is not consulted in the reverse branch of `_message_in_range`. So
|
||||
`offset_id` is an exclusive lower bound, the cursor stores the last handled id,
|
||||
and `iter_posts` carries a defensive `msg.id <= after_id` guard regardless.
|
||||
- All 15 error classes in phase 5's taxonomy exist in 1.44.0. A missing one would
|
||||
have been an import-time crash.
|
||||
- P2, P3, P5 and P6 remain live-only. The design was already answer-independent
|
||||
for all four, which is why implementation did not wait on them.
|
||||
|
||||
`spike/probe_telethon_api.py` is written and ready to run.
|
||||
|
||||
#### Module deviation
|
||||
|
||||
`media.py` was added, holding `media_kind` / `media_size` / `file_stem` /
|
||||
`file_ext`. The plan put `media_size` in `traversal.py`, but `paths.py` needs kind
|
||||
and extension for `derive_filename`, and either direction of import between those
|
||||
two would have been a cycle or a second copy of the rule. One owner, two
|
||||
consumers, no cycle.
|
||||
|
||||
One behavioral consequence worth recording: `media_kind` returns `None` when
|
||||
`msg.web_preview` is set. Telethon's `Message.photo` and `.document` also return
|
||||
the *web preview's* media, so classifying without that guard would have made every
|
||||
message containing a URL look like an attachment — inflating counts and
|
||||
downloading link-preview thumbnails.
|
||||
|
||||
#### Defects found and fixed during implementation
|
||||
|
||||
| # | Defect | Consequence had it shipped |
|
||||
|---|--------|---------------------------|
|
||||
| 1 | The 5-second commit throttle never flushed the tail | A run ending on a short stretch of filtered or text-only posts set `completed_at` against a stale cursor; the next run re-swept that tail to download nothing |
|
||||
| 2 | `git check-ignore` reports `exports` as *not ignored* until the directory exists | The export root would be refused on the first run and accepted on every run after it. Fixed with a directory-form retry, gated to `is_dir=True` so it can never excuse a session *file* |
|
||||
| 3 | `file_ext` capped extensions at 16 characters, not bytes | A multibyte suffix pushed the filename past ext4's 255-byte limit |
|
||||
|
||||
#### Corrections to the plan's own sketches
|
||||
|
||||
- **`except` ordering.** The plan checked `OSError` before the network clause, but
|
||||
`ConnectionError` *is* an `OSError`, and asyncio's `ConnectionResetError` can
|
||||
carry `errno=None` — which the errno test would have routed to "unexpected OS
|
||||
error" and aborted a healthy run. Network clause first, filesystem second.
|
||||
- **Lock contention exit code.** `raise SystemExit(f"busy: ...")` exits **1**, not
|
||||
3, so it would have failed its own acceptance criterion. Raises `Abort(reason, 3)`.
|
||||
- **Exit-code ownership.** `run_download` lets `Abort` and `KeyboardInterrupt`
|
||||
propagate to `cli.main()` rather than calling `sys.exit` mid-loop, and does not
|
||||
call `disconnect()` — the `connected_client` context manager owns that.
|
||||
- **Bare asserts.** The ascending-sweep and album-buffer tripwires are now
|
||||
`SweepOrderError`, not `assert`: `python -O` strips assertions, and these guard
|
||||
the same silent corruption as the `AlbumSplitError` beside them.
|
||||
|
||||
### Red Team Review — Session 2 — 2026-08-13 (post-implementation)
|
||||
|
||||
Independent `code-reviewer` pass over the implementation against this plan.
|
||||
**Verdict: DONE_WITH_CONCERNS.** Invariants 1 and 2 verified to hold under trace
|
||||
and probe; dry-run and real-run agree exactly on posts and files for
|
||||
`limit ∈ {None, 0, 1, 2}`. All eight declared deviations were assessed sound and
|
||||
kept. Two reachable defects broke stated guarantees:
|
||||
|
||||
| # | Finding | Severity | Disposition |
|
||||
|---|---------|----------|-------------|
|
||||
| H1 | `--types " "` (or `,`) builds an empty kind set that drops every media kind, but `to_state()` serialized it as `None` — i.e. *no filter* | Critical | **Fixed.** Refused in `build_filters`; `to_state` made lossless as a second line of defense |
|
||||
| H2 | The filename **extension** bypassed `sanitize` entirely | Critical | **Fixed.** Routed through `sanitize_ext`; plus `download_one` now records a path failure as FAILED instead of letting it escape |
|
||||
| M1 | `validate` excused *any* photo size mismatch, including 0 bytes | High | **Fixed.** Zero bytes is refused for every kind |
|
||||
| M2 | A FAILED file advances the cursor and is never retried | Medium | **Documented, not changed** — the plan settles this ("recorded and skipped, the run continues"). See open question 9 |
|
||||
| M3 | `ServerError` (parent of `RpcCallFailError`) and the `telethon.errors.common` set escaped the taxonomy | High | **Fixed.** Both added to the retry tuple |
|
||||
| M4 | `OSError` from the sidecar or state writes bypassed the exit-code contract | Medium | **Fixed.** `cli.main` maps `FATAL_ERRNO` → 3 |
|
||||
| M5 | `--max-size 0` printed as `(none)`; a negative `--limit` made the two run modes disagree | Low | **Fixed.** `is not None` throughout; negative `--limit` rejected at the boundary |
|
||||
| M6 | `Config.max_flood_wait` was dead | Low | **Fixed.** Removed |
|
||||
|
||||
H1 was the worst of these: it is silent *total* data loss, reachable from
|
||||
`--types "$UNSET_VAR"`, and it defeats Invariant 3 — the very guard meant to catch
|
||||
it. H2 was a permanent denial of service from one hostile filename, since the
|
||||
cursor could not advance past the message that raised.
|
||||
|
||||
Also fixed from the review's lower-severity notes: duplicated `_fsync_dir`
|
||||
(now `state.fsync_dir`, one owner), the session file being created before the git
|
||||
check rather than after, dead code in `human_bytes` and `EstimateSink.add`, and a
|
||||
dry-run `Mode:` line that silently measured from zero when the stored filters
|
||||
differed — it now says a real run would refuse.
|
||||
|
||||
Two criteria the review found untested are now tested: export-tree and file modes
|
||||
under the process umask, and control characters in the *extension*.
|
||||
|
||||
## Validation Log
|
||||
|
||||
### Session 1 — 2026-08-11
|
||||
|
||||
Verification pass skipped per the workflow guard: `## Red Team Review` above already carries evidence-backed verification, and zero `[UNVERIFIED]` tags remain across all eight plan files.
|
||||
|
||||
**Questions asked:** 5 (4 + 1 disambiguation)
|
||||
|
||||
| # | Decision point | Answer | Effect |
|
||||
|---|---|---|---|
|
||||
| 1 | Export directory naming | `g<chat_id>` + `title.txt` | **Confirms** the red-team fix. No change. |
|
||||
| 2 | Media with no reported size | Download + warn | **Confirms** the resolved design. No change. |
|
||||
| 3 | Sidecar scope | Add an `--include-text` flag | **Change** — see below |
|
||||
| 4 | Disk reserve | 2 GiB | **Confirms.** No change. |
|
||||
| 5 | Flood-wait ceiling | **No ceiling by default**; `--max-flood-wait N` caps it | **Change** — reverses the plan's prior default |
|
||||
|
||||
**Change 1 — `--include-text`.** Default remains media-bearing only. With the flag, text-only messages are recorded in the sidecar with an empty `files[]`. It joins the stored filter set because it changes what a post contains: running without it and then with it against the same cursor would otherwise silently skip the text records. Text-only messages never create a post folder and never affect `posts_with_files` counting, so the dry-run/real-run reconciliation is unaffected.
|
||||
|
||||
**Change 2 — flood-wait default inverted.** Previously the tool exited resumably after a wait beyond 1 hour. Now it sleeps indefinitely by default, and `--max-flood-wait N` imposes a ceiling.
|
||||
|
||||
The trade-off was presented and the user chose unattended completion over fail-fast: a badly throttled account now looks like a hang rather than an exit. Accepted with one mitigation — every flood wait logs at WARNING with its duration **and computed wake time**, so a long sleep is legible as a sleep. Exit code 6 is retained for the capped case.
|
||||
|
||||
### Whole-Plan Consistency Sweep — Validation Session 1
|
||||
- Files reread: plan.md, phase-01 … phase-07
|
||||
- Decision deltas checked: 2
|
||||
- Reconciled: flag surface (plan/ph2/ph3/ph5), filter set membership of `--include-text` (ph3/ph5), sidecar record shape for text messages (ph5), `posts_with_files` counting (ph3/ph6), flood-wait defaults and exit code 6 (plan/ph2/ph5), README items (ph7)
|
||||
- Unresolved contradictions: 0
|
||||
+325
@@ -0,0 +1,325 @@
|
||||
# Research Report: Telegram Group Media Exporter (Telethon / Python)
|
||||
|
||||
**Conducted:** 2026-08-11 14:11 (Asia/Saigon)
|
||||
**Target env:** Linux ARM64 (aarch64), headless, Python 3.12.3, ffmpeg 6.1.1, 41 GB free disk
|
||||
**Stack (confirmed by user, not re-evaluated):** Telethon + Python, album-aware folder layout
|
||||
**Scope in v1:** resume+dedupe, `messages.jsonl` sidecar, type/date/size filters, dry-run size estimate
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Executive Summary](#executive-summary)
|
||||
2. [Methodology](#methodology)
|
||||
3. [Findings](#findings)
|
||||
- [1. Auth & Session Persistence](#1-auth--session-persistence)
|
||||
- [2. History Traversal & Resume](#2-history-traversal--resume)
|
||||
- [3. Album Grouping via grouped_id](#3-album-grouping-via-grouped_id)
|
||||
- [4. Media Download & Filenames](#4-media-download--filenames)
|
||||
- [5. Flood Waits & Concurrency](#5-flood-waits--concurrency)
|
||||
- [6. Dry-Run Size Estimation](#6-dry-run-size-estimation)
|
||||
- [7. Packaging & CLI](#7-packaging--cli)
|
||||
4. [Proposed Architecture](#proposed-architecture)
|
||||
5. [Common Pitfalls](#common-pitfalls)
|
||||
6. [Confidence Ledger](#confidence-ledger)
|
||||
7. [Next Steps](#next-steps)
|
||||
8. [Unresolved Questions](#unresolved-questions)
|
||||
9. [References](#references)
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Three findings drive the whole design.
|
||||
|
||||
**Concurrency is the wrong instinct.** Telethon does not parallelize file downloads internally, and the docs/issues are explicit that parallel downloads only make `FloodWaitError` arrive *sooner*. Worse, retrying during an active flood wait is itself counted as a violation — a 7s wait can escalate past 4 hours within an hour of retry-looping. **Default concurrency must be 1**, with explicit flood-wait sleeping. This kills any worker-pool design before it is written.
|
||||
|
||||
**Telethon has no byte-level download resume.** `download_media` writes from offset 0 every call. Resumability must therefore be built at *message granularity*: download to a `.part` file, `os.replace()` atomically on completion, and treat any surviving `.part` as incomplete garbage to be re-fetched. Combined with an oldest→newest sweep (`reverse=True`) and a persisted last-completed message id, this gives a monotonic, crash-safe cursor.
|
||||
|
||||
**Albums need buffering, not lookups.** An album is N separate messages sharing a `grouped_id`; there is no "fetch the album" API. The common StackOverflow pattern (scan ±10 message ids around a post) is for the single-post case and is wasteful here. Since we sweep every message anyway, group by buffering consecutive same-`grouped_id` messages and flushing when the id changes.
|
||||
|
||||
---
|
||||
|
||||
## Methodology
|
||||
|
||||
- Sources consulted: 5 research passes (1 doc fetch, 4 web searches) → ~20 distinct sources
|
||||
- Gemini CLI: **absent on this host**, fell back to WebSearch per skill config
|
||||
- Primary authority: `docs.telethon.dev` (1.44.0 stable) + LonamiWebs/Telethon GitHub issues
|
||||
- Key terms: `grouped_id album`, `FloodWaitError bulk download`, `StringSession vs SQLiteSession`, `iter_messages reverse offset_id`, `pyproject.toml CLI 2026`
|
||||
- **Limitation:** `docs.telethon.dev/en/stable/modules/client.html` is a single very large page; the fetch truncated before the full `iter_messages` and `download_media` signatures. Items depending on it are flagged in the [Confidence Ledger](#confidence-ledger).
|
||||
|
||||
---
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. Auth & Session Persistence
|
||||
|
||||
Bot API is structurally disqualified (no history access; 20 MB download cap) — MTProto user account is the only path. `api_id`/`api_hash` from my.telegram.org.
|
||||
|
||||
| | SQLiteSession (default) | StringSession |
|
||||
|---|---|---|
|
||||
| Storage | on-disk `.session` SQLite DB | in-memory, serializable to base64 string |
|
||||
| Best for | long-lived local tool | serverless / env-var-injected secrets |
|
||||
| Gotcha | **must call `disconnect()` before exit** or state is not flushed | must persist the string yourself |
|
||||
|
||||
**Recommendation: SQLiteSession file**, `chmod 600`, path configurable, `.gitignore`d. This is a localhost tool with a persistent filesystem — StringSession solves a containerization problem we do not have (YAGNI). One-time interactive login (phone + code), every later run unattended.
|
||||
|
||||
Security, verbatim from the docs: *"Never give the saved session file to anyone, since they would gain instant access to all your messages and contacts."* Treat `.session` as a credential — never commit, never log its path contents.
|
||||
|
||||
**2FA is unresolved by the sources.** The searched docs did not cover the 2FA password flow. Telethon's `client.start(phone=..., password=...)` and `SessionPasswordNeededError` are the expected mechanism — **verify at implementation time**, and never accept the 2FA password as a CLI argument (it lands in shell history); prompt interactively or read from env.
|
||||
|
||||
### 2. History Traversal & Resume
|
||||
|
||||
`iter_messages(entity, ...)` with parameters `limit`, `offset_id`, `min_id`, `max_id`, `offset_date`, `filter`, `reverse` — all confirmed present, full semantics truncated in the fetch.
|
||||
|
||||
**Design: sweep oldest→newest with `reverse=True`.** Rationale: message ids increase monotonically over time, so "highest fully-completed id" is a single-integer resume cursor that is always correct, even if the run is killed mid-album. A newest-first sweep would need an interval set instead.
|
||||
|
||||
```python
|
||||
# resume: continue after the last fully-completed message
|
||||
async for msg in client.iter_messages(entity, reverse=True, offset_id=state.last_done_id):
|
||||
...
|
||||
```
|
||||
|
||||
**Footgun to verify:** `offset_id` semantics *invert* with `reverse`. Newest-first, it is an upper bound (start below this id); with `reverse=True` it acts as a lower bound (start above this id). Confirm empirically with a 3-message probe before trusting the resume path — an off-by-one here silently re-downloads or silently skips.
|
||||
|
||||
Pagination is handled internally (~100 messages/request); no manual paging needed.
|
||||
|
||||
### 3. Album Grouping via `grouped_id`
|
||||
|
||||
Confirmed: *"Albums are a way to send multiple photos or videos as separate messages with the same grouped identifier. It is possible to know that it's a grouped message thanks to `grouped_id`."*
|
||||
|
||||
Album members arrive as **consecutive** messages in a chronological sweep. Grouping algorithm:
|
||||
|
||||
```
|
||||
buffer = []
|
||||
for msg in sweep(reverse=True):
|
||||
if msg.grouped_id is None:
|
||||
flush(buffer); emit_single(msg)
|
||||
elif buffer and msg.grouped_id != buffer[0].grouped_id:
|
||||
flush(buffer); buffer = [msg]
|
||||
else:
|
||||
buffer.append(msg)
|
||||
flush(buffer) # end of history
|
||||
```
|
||||
|
||||
**Folder id = the lowest message id in the group** (first member, since we sweep ascending). Deterministic and resume-stable: a re-run recomputes the same folder name.
|
||||
|
||||
**Resume interacts with albums.** Only advance `last_done_id` once *every* member of a group has landed — otherwise a crash mid-album leaves a half-folder that resume skips past. Simplest correct rule: commit the cursor at group-flush boundaries, not per file.
|
||||
|
||||
Do **not** use the "iterate ±N ids around the post" pattern from issue #3788 — that solves the single-known-post case, and duplicates work in a full sweep.
|
||||
|
||||
### 4. Media Download & Filenames
|
||||
|
||||
Confirmed chunked download primitive:
|
||||
|
||||
```python
|
||||
async download_file(
|
||||
input_location: hints.FileLike,
|
||||
file: hints.OutFileLike = None,
|
||||
*,
|
||||
part_size_kb: float = None,
|
||||
file_size: int = None,
|
||||
progress_callback: hints.ProgressCallback = None,
|
||||
dc_id: int = None,
|
||||
key: bytes = None,
|
||||
iv: bytes = None
|
||||
) -> bytes | None
|
||||
```
|
||||
|
||||
For our purposes the higher-level `download_media(message, file=path, progress_callback=...)` is correct — it wraps chunking and DC routing. `part_size_kb` tuning is a micro-optimization; skip it (YAGNI).
|
||||
|
||||
**Filename/extension derivation:** `message.file` exposes `.name`, `.ext`, `.mime_type`, `.size`. Preference order:
|
||||
1. `DocumentAttributeFilename` via `message.file.name` — real filename for documents
|
||||
2. `message.file.ext` + mime type — for photos/voice/round video, which carry no filename
|
||||
3. Fallback `bin`
|
||||
|
||||
**Sanitize aggressively.** Filenames are attacker-controlled strings from arbitrary group members: strip `/`, `..`, NUL, leading dots, control chars, and cap length. Always join under the target dir and assert the resolved path stays inside it. A crafted `../../../.ssh/authorized_keys` is a real, cheap attack on a naive exporter.
|
||||
|
||||
**No byte-level resume** (see Executive Summary). Pattern:
|
||||
|
||||
```
|
||||
tmp = folder / f"{idx}_{name}.part"
|
||||
download_media(msg, file=tmp)
|
||||
os.replace(tmp, folder / f"{idx}_{name}") # atomic on same filesystem
|
||||
```
|
||||
|
||||
**Dedupe check:** skip when the final file exists *and* its size equals `message.file.size`. Size-match is cheap and catches truncation; hashing every file is wasted I/O at multi-GB scale.
|
||||
|
||||
### 5. Flood Waits & Concurrency
|
||||
|
||||
The single most decision-relevant finding.
|
||||
|
||||
- `client.flood_sleep_threshold` (default **60s**): Telethon auto-sleeps on flood waits ≤ threshold, raises `FloodWaitError` above it.
|
||||
- *"The library does not download or upload files in parallel"* — no internal parallelism to inherit.
|
||||
- *"The limiting factor in the long run are FloodWaitError, and using parallel download or uploads only makes them occur sooner."*
|
||||
- **Escalation is real:** *"a wait that starts at 7 seconds can escalate past four hours within less than an hour of repeated restarts, because every retry during an active FloodWait is itself treated as another violation."*
|
||||
|
||||
**Design rules:**
|
||||
1. Sequential downloads. Default concurrency **1**. Do not ship a `--workers` flag in v1 — it is a footgun that trades a small speedup for hour-long lockouts.
|
||||
2. Raise `flood_sleep_threshold` (e.g. 120s) so routine small waits are absorbed silently.
|
||||
3. Catch `FloodWaitError` explicitly, `await asyncio.sleep(e.seconds + jitter)`, then continue. **Never** tight-retry, never restart the process to "get around" a wait.
|
||||
4. Log every flood wait with its duration — it is the primary signal for whether the run is healthy.
|
||||
|
||||
### 6. Dry-Run Size Estimation
|
||||
|
||||
`message.file.size` gives bytes without fetching content. Dry-run = full `iter_messages` sweep with zero `download_media` calls: cheap (~100 messages/request, no file-DC traffic), and the flood-wait risk is low because history reads are far cheaper than file reads.
|
||||
|
||||
Output: total media count, total bytes, breakdown by type, and **free-disk delta**. With 41 GB free on `/config`, an export that does not fit should fail loudly *before* the first byte, not at 97%.
|
||||
|
||||
Caveat: for photos, `.size` reflects the size variant Telethon selects (largest by default); treat photo totals as a close estimate, document sizes as exact.
|
||||
|
||||
### 7. Packaging & CLI
|
||||
|
||||
Consensus from the packaging guides: `argparse` for simple tools with no dependencies, Click for complex multi-command CLIs, Typer for type-hint ergonomics (built on Click).
|
||||
|
||||
**Recommendation: `argparse`.** One command, ~8 flags, no subcommands. Adding Typer means +2 transitive deps to save perhaps 25 lines. YAGNI/KISS. If subcommands ever appear, Typer is a contained swap.
|
||||
|
||||
Packaging: `pyproject.toml` with `[project.scripts]` entry point (`tg-export = "telegram_exporter.cli:main"`) — the current standard, declarative and static.
|
||||
|
||||
**`uv` is not installed on this host.** Options: `pip install -e .` inside a venv (works today, zero setup), or install `uv` first. The venv path is fine; nothing here needs uv.
|
||||
|
||||
---
|
||||
|
||||
## Proposed Architecture
|
||||
|
||||
```
|
||||
┌──────────────┐
|
||||
│ cli.py │ argparse: --group --out --dry-run
|
||||
│ │ --since/--until --types --max-size
|
||||
└──────┬───────┘
|
||||
│
|
||||
┌──────▼───────┐
|
||||
│ client.py │ auth, SQLiteSession, flood-wait wrapper
|
||||
└──────┬───────┘
|
||||
│
|
||||
┌───────────────────▼────────────────────┐
|
||||
│ sweep.py │
|
||||
│ iter_messages(reverse=True, │
|
||||
│ offset_id=cursor) │
|
||||
└───────────────────┬────────────────────┘
|
||||
│ messages, ascending
|
||||
┌───────────────────▼────────────────────┐
|
||||
│ grouper.py buffer by grouped_id │
|
||||
└───────────────────┬────────────────────┘
|
||||
│ logical posts
|
||||
┌─────────────┴─────────────┐
|
||||
│ │
|
||||
┌────────▼────────┐ ┌────────▼────────┐
|
||||
│ estimator.py │ │ downloader.py │
|
||||
│ sum .file.size │ │ .part → rename │
|
||||
│ (dry-run only) │ │ size-match skip │
|
||||
└─────────────────┘ └────────┬────────┘
|
||||
│
|
||||
┌─────────────▼─────────────┐
|
||||
│ state.py cursor + jsonl │
|
||||
│ commit at group boundary │
|
||||
└───────────────────────────┘
|
||||
```
|
||||
|
||||
Output layout (album-aware, per user decision):
|
||||
|
||||
```
|
||||
exports/<group>/
|
||||
1042/ <- album, grouped_id shared, folder = first msg id
|
||||
0_photo.jpg
|
||||
1_photo.jpg
|
||||
2_video.mp4
|
||||
1047/ <- standalone message
|
||||
0_document.pdf
|
||||
messages.jsonl <- id, date, sender, caption, grouped_id, reply_to, files[]
|
||||
.export-state.json <- last_done_id, counters
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
| Pitfall | Consequence | Mitigation |
|
||||
|---|---|---|
|
||||
| Parallel downloads to "go faster" | Flood waits arrive sooner, escalate to hours | Sequential, concurrency 1 |
|
||||
| Retrying during an active flood wait | 7s wait → 4h+ lockout | Sleep the full `e.seconds`, never restart to bypass |
|
||||
| Forgetting `disconnect()` | SQLite session not flushed, re-login next run | `try/finally` or async context manager |
|
||||
| Unsanitized `DocumentAttributeFilename` | Path traversal outside the export dir | Sanitize + assert resolved path is inside target |
|
||||
| Advancing the resume cursor per file | Crash mid-album → permanently half-downloaded folder | Commit cursor only at group flush |
|
||||
| Assuming `download_media` resumes | Silent truncated files | `.part` + atomic rename; treat stray `.part` as garbage |
|
||||
| `offset_id` semantics under `reverse=True` | Silent skip or full re-download | Verify empirically with a 3-message probe |
|
||||
| Committing the `.session` file | Full account takeover | `.gitignore`, `chmod 600` |
|
||||
| No pre-flight size check | Fills 41 GB disk mid-run | Dry-run + free-space assertion |
|
||||
|
||||
---
|
||||
|
||||
## Confidence Ledger
|
||||
|
||||
Honest separation of what the sources confirmed from what needs checking in code.
|
||||
|
||||
**Verified by sources:**
|
||||
- `flood_sleep_threshold` default 60s, auto-sleep ≤ threshold
|
||||
- Telethon does not parallelize file transfers; parallelism accelerates flood waits
|
||||
- Flood-wait escalation from retries during an active wait
|
||||
- `download_file` full signature incl. `part_size_kb`, `progress_callback`, `dc_id`
|
||||
- `grouped_id` identifies album membership; albums are separate messages
|
||||
- SQLiteSession vs StringSession trade-offs; `disconnect()` flush requirement
|
||||
- `iter_messages` accepts `reverse`, `offset_id`, `min_id`, `max_id`, `offset_date`, `filter`, `limit`
|
||||
- `[project.scripts]` in `pyproject.toml` as the entry-point standard
|
||||
|
||||
**Not confirmed — verify during implementation:**
|
||||
- Exact `offset_id` direction under `reverse=True` (**highest-risk item**; breaks resume silently)
|
||||
- 2FA / `SessionPasswordNeededError` flow — sources did not cover it
|
||||
- `message.file` attribute surface (`.name`/`.ext`/`.size`/`.mime_type`) — strongly indicated by the custom-types docs, not read verbatim
|
||||
- Whether `download_media` returns the final path string in all media branches
|
||||
- Photo `.size` accuracy vs the actually-downloaded variant
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Spike first, build second** (~30 min): a throwaway script that logs in, iterates 20 messages of a real group, prints `id / grouped_id / file.name / file.size`, and downloads one album. Settles all four unverified items above at once.
|
||||
2. Run `/ck:plan` to phase the implementation against the architecture sketch.
|
||||
3. Phase order: auth+session → sweep+resume → grouper → downloader → jsonl sidecar → filters → dry-run estimator.
|
||||
4. Add `.session`, `exports/`, `.export-state.json` to `.gitignore` **before** the first login.
|
||||
|
||||
---
|
||||
|
||||
## Unresolved Questions
|
||||
|
||||
1. **Group scale?** Unknown message/media count. If the group is >40 GB the 41 GB free disk is the binding constraint and dry-run becomes mandatory before any real run.
|
||||
2. **2FA enabled on the account?** Changes the login flow and whether an interactive prompt is required on first run.
|
||||
3. **Group access type** — public/private, and is the account already a member? Private groups need the account joined; `get_entity` resolution differs for public username vs invite-link groups.
|
||||
4. **Should non-media messages appear in `messages.jsonl`?** Text-only messages give context to captions and replies, but inflate the sidecar. Default assumption: media-bearing messages only.
|
||||
5. **Deleted/expiring media and service messages** — skip silently or record as gaps in the sidecar?
|
||||
6. **Retention of `.part` files** on a failed run — auto-clean at startup, or leave for inspection?
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
### Official Documentation
|
||||
- [Client class — Telethon 1.44.0](https://docs.telethon.dev/en/stable/modules/client.html) — `download_file` signature, `iter_messages` params
|
||||
- [RPC Errors — Telethon 1.44.0](https://docs.telethon.dev/en/stable/concepts/errors.html) — FloodWaitError semantics
|
||||
- [FAQ — Telethon 1.44.0](https://docs.telethon.dev/en/stable/quick-references/faq.html) — no-parallel-transfer statement
|
||||
- [Session Files — Telethon 1.44.0](https://docs.telethon.dev/en/stable/concepts/sessions.html) — SQLite vs String sessions
|
||||
- [Sessions module — Telethon 1.44.0](https://docs.telethon.dev/en/stable/modules/sessions.html)
|
||||
- [Custom package (types) — Telethon 1.44.0](https://docs.telethon.dev/en/stable/modules/custom.html) — `Message.file` wrapper
|
||||
- [Telethon 1.44.0 full PDF](https://media.readthedocs.org/pdf/telethon/stable/telethon.pdf) — offline reference for the truncated pages
|
||||
- [Creating and packaging command-line tools — PyPA](https://packaging.python.org/en/latest/guides/creating-command-line-tools/)
|
||||
- [Writing your pyproject.toml — PyPA](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/)
|
||||
|
||||
### Issues & Community
|
||||
- [Download all media in a post by grouped_id — Telethon #3788](https://github.com/LonamiWebs/Telethon/issues/3788)
|
||||
- [Detect a grouped message (album) — Telethon #1216](https://github.com/LonamiWebs/Telethon/issues/1216)
|
||||
- [FloodWaitError in client.download_media — Telethon #1426](https://github.com/LonamiWebs/Telethon/issues/1426)
|
||||
- [FloodWaitError during document download — Telethon #206](https://github.com/LonamiWebs/Telethon/issues/206)
|
||||
- [Converting StringSession to SQLiteSession — Telethon #3660](https://github.com/LonamiWebs/Telethon/issues/3660)
|
||||
- [Telethon — DeepWiki](https://deepwiki.com/LonamiWebs/Telethon)
|
||||
|
||||
### Further Reading
|
||||
- [Fix Telegram FloodWait Error Fast](https://membertel.com/blog/how-to-fix-telegram-floodwait-error-fast/) — flood-wait escalation behavior
|
||||
- [telebackup — multi-connection Telegram downloader](https://github.com/xwc9527/telebackup) — prior art for parallel/checkpoint downloading; **contrary to our sequential decision**, review only if throughput becomes a proven problem
|
||||
- [Packaging Python CLI apps the modern way with uv](https://thisdavej.com/packaging-python-command-line-apps-the-modern-way-with-uv/)
|
||||
- [Building Python CLI tools: Click, Typer, argparse](https://inventivehq.com/blog/python-cli-tools-guide)
|
||||
|
||||
### Glossary
|
||||
- **MTProto** — Telegram's native protocol; full user-account API access (vs the restricted Bot API)
|
||||
- **DC** — Telegram data center; large files may live on a DC other than the session's home DC
|
||||
- **`grouped_id`** — shared identifier marking messages that belong to one album
|
||||
- **FloodWait** — server-imposed cooldown after too-rapid requests; escalates on retry
|
||||
@@ -0,0 +1,139 @@
|
||||
---
|
||||
title: "Phase 1 — Telethon API Verification Findings"
|
||||
phase: 1
|
||||
status: partial
|
||||
telethon_version: "1.44.0"
|
||||
created: "2026-08-12"
|
||||
---
|
||||
|
||||
# Phase 1 — Telethon API Verification Findings
|
||||
|
||||
Two sources of evidence appear below and are labelled separately:
|
||||
|
||||
- **SOURCE** — settled offline by reading the installed `telethon` 1.44.0 package.
|
||||
Deterministic, and stronger than a single live observation for questions about
|
||||
parameter semantics, because it shows the code path rather than one sample of
|
||||
its behavior.
|
||||
- **LIVE** — requires `TG_API_ID`/`TG_API_HASH`, an interactive login, and a real
|
||||
group. **Not yet run.** Run `spike/probe_telethon_api.py` and paste its output
|
||||
into the corresponding section.
|
||||
|
||||
Environment: Python 3.12.3, aarch64, Telethon **1.44.0** (wheel, no build step),
|
||||
pinned in `requirements.txt`.
|
||||
|
||||
## Confidence ledger
|
||||
|
||||
| Probe | Question | Status | Answer |
|
||||
|---|---|---|---|
|
||||
| P1 | `offset_id` direction/inclusivity under `reverse=True` | **CONFIRMED (SOURCE)** | Exclusive lower bound: results satisfy `id > offset_id` |
|
||||
| P2 | 2FA / login flow | **PENDING (LIVE)** | — |
|
||||
| P3 | `message.file` surface; which kinds lack `.size` | **PENDING (LIVE)** | — |
|
||||
| P4 | `download_media` return value; file-object support | **PARTIAL (SOURCE)** | File objects are accepted; return value pending live confirmation |
|
||||
| P5 | photo declared `.size` vs bytes on disk | **PENDING (LIVE)** | — |
|
||||
| P6 | album adjacency across a pagination boundary | **PENDING (LIVE)** | Runtime tripwire implemented regardless |
|
||||
| — | error classes in phase 5's taxonomy exist | **CONFIRMED (SOURCE)** | All 15 present |
|
||||
|
||||
## P1 — `offset_id` under `reverse=True` — CONFIRMED (SOURCE)
|
||||
|
||||
`telethon/client/messages.py`, `_MessagesIter._init`:
|
||||
|
||||
```python
|
||||
if self.reverse:
|
||||
if offset_id:
|
||||
offset_id += 1
|
||||
elif not offset_date:
|
||||
offset_id = 1
|
||||
```
|
||||
|
||||
and `_update_offset`, run after every page:
|
||||
|
||||
```python
|
||||
self.request.offset_id = last_message.id
|
||||
if self.reverse:
|
||||
self.request.offset_id += 1 # "We want to skip the one we already have"
|
||||
```
|
||||
|
||||
**`offset_id` is an exclusive lower bound under `reverse=True`.** `offset_id=N`
|
||||
yields messages with `id > N`; `N` itself is excluded. `offset_id=0` starts from
|
||||
the beginning of history.
|
||||
|
||||
`min_id` was compared as the plan asked: `_message_in_range`'s `reverse` branch
|
||||
tests `message.id <= self.last_id or message.id >= self.max_id` and **does not
|
||||
consult `min_id` at all**, so `offset_id` is the correct parameter for a
|
||||
chronological sweep, not `min_id`.
|
||||
|
||||
**Consequence, implemented:** the cursor stores the **last handled id** and is
|
||||
passed as `offset_id`. `iter_posts` additionally carries an explicit
|
||||
`if msg.id <= after_id: continue` guard, so the sweep stays correct even if this
|
||||
internal detail changes in a future Telethon release —
|
||||
`src/telegram_exporter/traversal.py`.
|
||||
|
||||
## Error-class inventory — CONFIRMED (SOURCE)
|
||||
|
||||
Every exception phase 5's taxonomy catches exists in 1.44.0 (checked against
|
||||
`telethon.errors` and `telethon.errors.rpcerrorlist`). A missing name would have
|
||||
been an import-time crash on the first run:
|
||||
|
||||
`FloodWaitError`, `FileReferenceExpiredError`, `MediaEmptyError`,
|
||||
`FileIdInvalidError`, `FilerefUpgradeNeededError`, `AuthKeyUnregisteredError`,
|
||||
`AuthKeyDuplicatedError`, `SessionRevokedError`, `UserDeactivatedBanError`,
|
||||
`ChannelPrivateError`, `ChatForbiddenError`, `ChannelInvalidError`,
|
||||
`TimedOutError`, `RpcCallFailError`, `SessionPasswordNeededError`.
|
||||
|
||||
Constructor signature for all of them is `(request)`, with `FloodWaitError`
|
||||
taking `(request, capture=0)` where `capture` becomes `.seconds`. The test suite
|
||||
constructs them this way.
|
||||
|
||||
## P4 — `download_media` file-object support — PARTIAL (SOURCE)
|
||||
|
||||
`Message.file` wraps `photo or document` and is `None` for other media types
|
||||
(polls, games, none), which is why `media.media_kind()` classifies from the media
|
||||
type rather than from `.file`.
|
||||
|
||||
Telethon accepts a stream for `file=`, so phase 5 holds the fd and `fsync`s it
|
||||
before `os.replace`. The plan's fallback — download to a path, then re-open and
|
||||
fsync — is not needed unless the live probe contradicts this.
|
||||
|
||||
**Still pending (LIVE):** the exact return type/value of
|
||||
`download_media(msg, file=<path>)`.
|
||||
|
||||
## P3 / P5 note — why the implementation does not wait on these
|
||||
|
||||
Both remaining questions are about *sizes*, and the resolved design is
|
||||
answer-independent by construction:
|
||||
|
||||
- `media.media_size()` returns `int | None`, and each of the four consumers has a
|
||||
stated rule for `None` (include+warn / accept / separate tally / treat as 0).
|
||||
Tested in `tests/test_traversal.py`.
|
||||
- The download validator never raises: a photo mismatch is accepted as a size
|
||||
variant, and a document/video/audio mismatch retries then reports FAILED.
|
||||
Tested in `tests/test_download_decisions.py`.
|
||||
|
||||
P3's answer improves the `--max-size` warning's accuracy; P5's improves the
|
||||
dry-run label and phase 7's reconciliation tolerance. Neither can invalidate code
|
||||
that already treats `None` as a first-class case.
|
||||
|
||||
## P2 / P6 — PENDING (LIVE)
|
||||
|
||||
- **P2** — record whether `SessionPasswordNeededError` is raised, and confirm a
|
||||
second run is non-interactive. The 2FA password is `getpass`-only, enforced by
|
||||
`tests/test_logging.py`.
|
||||
- **P6** — the probe requires ≥250 messages swept and asserts it rather than
|
||||
passing vacuously. Coverage is probabilistic either way; the deterministic
|
||||
guarantee is the runtime tripwire in `traversal._close`, which raises
|
||||
`AlbumSplitError` when a `grouped_id` reopens after its run closed. Tested for
|
||||
both `A B A` and `A B C A`.
|
||||
|
||||
## Cleanup owed after running the spike
|
||||
|
||||
Deleting `spike/probe.session` does **not** revoke server-side authorization.
|
||||
Terminate the probe session in Telegram → Settings → Devices. This is phase 7
|
||||
step 13 and the most-missed item, because the local file looks like the whole
|
||||
credential.
|
||||
|
||||
## Unresolved
|
||||
|
||||
1. P2, P3, P5 and P6 need a live run — no Telegram credentials exist in this
|
||||
environment.
|
||||
2. If P3 finds a media kind whose `.size` is absent *and* commonly large, revisit
|
||||
only the estimate's `unknown size` line; no code change is implied.
|
||||
Reference in new issue
Block a user