tiennm99
19837ceb85
rclone does not address a file by the bytes os.OpenFile wrote. Names handed to an Fs go through the backend encoder and names listed back are re-encoded to the standard set, neither of which the write path performs. A filename holding one of the rewritten characters was therefore stored under one string and looked up under another: confirmed against the local backend, where a written "a‛b.jpg" reports object not found and a written "a\nb.jpg" lists back as "a␊b.jpg". That is the same two-derivation divergence this program was written to remove, with rclone's encoder standing where filenamify used to. Such names are rejected, not encoded, for the same reason every other name is. The length limit ignored the ".part" suffix that is opened first, so a name just inside NAME_MAX passed the check and then failed to open on every pass, stalling the walk on that message forever. The suffix now lives beside the limit that has to account for it. filter.NewFilter(nil) does not build a neutral filter; it copies the package global, which rclone has already filled from RCLONE_*. Indexing inherited the operator's environment, so a stray RCLONE_MIN_SIZE emptied the index and re-downloaded the archive. Every narrowing field is now set explicitly and the result is asserted inactive. A subprocess test covers it, since the env is read at package init and t.Setenv is too late to observe anything. An ErrorDirNotFound from a subdirectory was also treated as an empty destination, returning a partial index as authoritative. t.me/c/<id> and t.me/s/<name> were passed through whole, and gotd reads the first path component as the username — resolving "c" or "s", which is a confusing failure at best and someone else's chat at worst, since one-character usernames exist. Both now yield the chat, and t.me/s/<name>/<id> is refused like any other message link. Two tests asserted the old behaviour. core's dcpool.Takeout deadlocks when takeout init fails: it holds the pool mutex and recovers by calling Client, which takes the same non-reentrant mutex. Telegram returns TAKEOUT_INIT_DELAY for a takeout started recently and takeout is on by default, so two runs in succession hang the process with no output and no response to cancellation. The session is established once here instead, falling back to a plain client, and the pool's own Takeout is never called.
telegram-exporter
Archive a Telegram chat's media to any rclone remote — S3, Google Drive, Dropbox, Backblaze B2, SFTP, WebDAV, pikpak, or anything else rclone supports — using far less local disk than the chat's total size.
tgexport embeds tdl and
rclone as libraries and runs both halves in one process.
Files are downloaded into a small staging directory and uploaded the moment each
one finishes, so local disk only ever holds what is in flight. A multi-terabyte
chat archives fine on a small disk. Telegram caps a single file at 2 GB (4 GB
from premium uploaders), so a few dozen GB of staging covers the worst case
regardless of chat size.
This exists because tdl can only write to a local directory — it has no
remote destination of any kind (tdl dl -d takes a filesystem path; tdl --storage is its session database, not an output target). tdl downloads over
MTProto with a user account, so Bot API limits do not apply: full history is
readable and there is no 20 MB download cap.
Requirements
- Go 1.25+ to build, or a prebuilt binary.
- tdl — only for
tdl login. https://docs.iyear.me/tdl/getting-started/installation/ - rclone — only to configure a remote. https://rclone.org/install/
Neither tool is invoked at run time; tgexport reads the session and config
they write.
Setup
Both steps are one-time.
tdl login # writes the Telegram session tgexport reads
rclone config # define the destination remote
go build -o tgexport ./cmd/tgexport
./tgexport doctor -r myremote:archive
doctor proves both halves work before a long run: it prints the logged-in
account, resolves the destination, and reports free space.
Build variants
| Build | Backends | Size |
|---|---|---|
go build ./cmd/tgexport |
every rclone backend | ~92 MB |
go build -tags slim ./cmd/tgexport |
pikpak only | ~49 MB |
A backend that is not compiled in does not exist at run time, so use the default build unless the destination will never change.
Usage
./tgexport sync -c CHAT -r REMOTE:PATH [options]
CHAT accepts a numeric id as printed by tdl chat ls, a username with or
without @, or a t.me/tg:// link. A Bot API -100… id is converted
automatically. A link to a single message is refused — it names a message, not
a chat.
# archive a chat, capping staging at 40 GiB
./tgexport sync -c @mychannel -r gdrive:telegram/media -m 40G
# check completeness without downloading anything
./tgexport verify -c @mychannel -r gdrive:telegram/media
# list what the chat holds
./tgexport list -c @mychannel
Options
| Flag | Default | Meaning |
|---|---|---|
-c |
— | chat id, username, or link (required) |
-r |
— | rclone destination, REMOTE:PATH (required) |
-d |
./staging |
staging directory for files in flight |
-m |
no cap | cap staging at a size, e.g. 40G |
--threads |
4 | connections per file |
--limit |
2 | files downloading at once |
--uploads |
2 | files uploading at once |
--min-free |
5 | stop if the remote has fewer than this many GiB free |
--limit-items |
0 | stop after N files; for smoke tests |
--confirm |
true | re-state each uploaded file to prove its size |
--takeout |
true | use a takeout session |
-n |
default |
tdl session namespace |
Exit codes
| Code | Meaning |
|---|---|
| 0 | complete |
| 1 | ran, but files remain |
| 2 | usage error |
| 3 | remote or Telegram failure |
| 130 / 143 | interrupted (SIGINT / SIGTERM) |
How it works
Re-running is the resume path. Each item is checked against a listing of the remote immediately before download, so an interrupted run picks up where it left off and a completed one downloads nothing.
Filenames. Every file is stored as {DialogID}_{MessageID}_{FileName}, where
FileName is exactly what Telegram reports. One function derives that string,
and the same string is used both to ask whether the file is already archived and
to write it — so the two can never disagree.
That last point is the reason this program exists. Its predecessor derived the
name twice: tdl chat export wrote the raw name into a JSON, while tdl dl
rendered it through a template applying filenamify, which rewrites characters a
filesystem rejects and collapses runs of !. A file whose name contained !!
was looked up under one name and stored under another, so the verifier never
found it and re-fetched it on every pass — forever, at 966 MB a time.
Note the consequence: names are not run through filenamify, so they are not
byte-compatible with what the old shell pipeline wrote. A file it stored under a
rewritten name will not be recognised and gets fetched again.
Disk. -m is a byte budget. A download reserves its own size before starting
and releases it only once the upload is confirmed, so when the remote is slow the
downloads pause on their own. The cap must exceed the largest single file, and a
cap that does not is refused at startup rather than discovered as a hang.
Integrity. A download is written to <name>.part and renamed only once its
size matches what Telegram reported, so a file without the suffix is always
whole. Uploads are re-stated afterwards to prove they arrived at the right size,
before the local copy is gone.
Replacing the shell pipeline
Earlier versions of this repo were three bash scripts — run.sh,
export-until-complete.sh and verify-export.sh — driving tdl and rclone as
separate processes. Everything expensive in them existed to work around the fact
that neither process could see the other's state: a staging directory polled with
du -sk, an --min-age guard, a *.tmp exclusion, SIGSTOP/SIGCONT to
enforce the disk cap, a sweep-failure counter, and an outer loop that re-verified
and re-narrowed a JSON export between passes.
One process needs none of it. Completion is a function returning; the cap is a semaphore. Some hard-won details were worth keeping, and are:
- pikpak commits uploads as a server-side async task, and rclone abandons a
still-pending one when its low-level retries run out.
transfers=2andlow-level-retries=20are the defaults here for that reason. Environment overrides still win. - A backend with no quota API is treated as unlimited, so it never blocks a run.
- Zero-byte files count as missing — rclone overwrites a size-mismatched destination, so re-running repairs them — while files under 1 KiB are reported but trusted, since some real media genuinely is that small.
Flags that disappeared are recognised and explain what replaced them:
| Old | Why it is gone |
|---|---|
-i |
no sweep interval; uploads start when a download finishes |
-a |
no --min-age; completion is observed, not inferred |
-f |
no export JSON; the chat is read live, so names cannot go stale |
-p |
no passes; one invocation converges |
-q |
renamed --min-free |
Notes
tgexportand thetdlCLI share one session store and cannot run against the same namespace at once. Use-nfor a second namespace if you need both.- A partially downloaded file is not resumable across restarts — tdl's library
exposes no resume offset — so an interrupted run re-fetches whatever was in
flight, bounded by
--limit. - Everything is read-only against Telegram. Nothing is uploaded, deleted, or marked read.