mirror of
https://github.com/tiennm99/export-telegram-group-members.git
synced 2026-10-15 18:15:54 +00:00
feat: port the tool to Go
Replace the Python scripts with Go commands run as go run ./cmd/<name>: configure, crawl, compare, compare-groups, and export, each with the same arguments and output as before. Telegram access uses gotd/td and the code is pure Go. Redis keys and JSON keep the Python format, so existing crawls stay readable. The Telethon login session is converted once into a separate gotd session key, so upgrading needs no new login code and never changes the old key. Tests run offline with miniredis.
This commit is contained in:
tiennm99 committed
1 parent
73f8cf1bb3
commit
d041503a47
51 files changed
+2684
-1551
No files matched your search
+1
-9
@@ -1,9 +1 @@
|
||||
# TG_PHONE is phone number in international format.
|
||||
# Like +4123456789.
|
||||
TG_PHONE=your_telegram_phone_number
|
||||
|
||||
# APP_HASH, APP_ID is from https://my.telegram.org/.
|
||||
APP_ID=your_app_id
|
||||
APP_HASH=your_app_hash
|
||||
|
||||
GROUP_IDS=-1001234567890
|
||||
REDIS_URL=rediss://default:<password>@<host>:<port>
|
||||
+10
-26
@@ -1,36 +1,20 @@
|
||||
*.sqlite3
|
||||
# project
|
||||
*.csv
|
||||
output/
|
||||
plans/
|
||||
|
||||
|
||||
|
||||
# If you prefer the allow list template instead of the deny list, see community template:
|
||||
# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
|
||||
#
|
||||
# Binaries for programs and plugins
|
||||
# go
|
||||
*.exe
|
||||
*.exe~
|
||||
*.dll
|
||||
*.so
|
||||
*.dylib
|
||||
|
||||
# Test binary, built with `go test -c`
|
||||
*.test
|
||||
|
||||
# Code coverage profiles and other test artifacts
|
||||
*.out
|
||||
coverage.*
|
||||
*.coverprofile
|
||||
profile.cov
|
||||
|
||||
# Dependency directories (remove the comment below to include it)
|
||||
# vendor/
|
||||
|
||||
# Go workspace file
|
||||
go.work
|
||||
go.work.sum
|
||||
|
||||
# env file
|
||||
# env
|
||||
.env
|
||||
|
||||
# Editor/IDE
|
||||
# .idea/
|
||||
# .vscode/
|
||||
# tooling
|
||||
.idea/
|
||||
.vscode/
|
||||
.claude/
|
||||
@@ -1,3 +1,166 @@
|
||||
# Export Telegram group members
|
||||
# export-telegram-group-members
|
||||
|
||||
*In 2025, I rewrote this project using Go. The original Python version of this project can be found at `feature/python` branch.*
|
||||
Export Telegram group members with Go and [gotd/td](https://github.com/gotd/td).
|
||||
Admin rights give full member visibility. The login session and export history
|
||||
live in **Redis**, so you can run the tool on any device from just a `.env`
|
||||
file: no re-login and no copying session or CSV files.
|
||||
|
||||
## How to use
|
||||
|
||||
1. Clone this repository and install [Go](https://go.dev/dl/):
|
||||
|
||||
```bash
|
||||
git clone https://github.com/tiennm99/export-telegram-group-members.git
|
||||
```
|
||||
|
||||
2. Create a new Telegram app at [https://my.telegram.org](https://my.telegram.org) and get the `api_id` and `api_hash`.
|
||||
3. Create a free Redis database (e.g. [Upstash](https://upstash.com)) and copy its `rediss://` connection URL.
|
||||
4. Copy `.env.example` to `.env` and fill in `REDIS_URL`.
|
||||
5. Store Telegram config in Redis:
|
||||
|
||||
```bash
|
||||
go run ./cmd/configure
|
||||
```
|
||||
|
||||
`configure` prompts for `api_id`, `api_hash`, phone, and group IDs. You can
|
||||
pass any of them as flags instead: `--api-id`, `--api-hash`, `--phone`, and
|
||||
`--groups -1001234567890,-1009876543210`. All commands read the stored group
|
||||
IDs from this Redis config.
|
||||
|
||||
6. Crawl the configured groups:
|
||||
|
||||
```bash
|
||||
go run ./cmd/crawl
|
||||
```
|
||||
|
||||
Every group ID stored in the Redis config is crawled. To change the list, run
|
||||
`configure` again.
|
||||
|
||||
To crawl other groups just once, pass their IDs; they replace the configured
|
||||
list for that run only:
|
||||
|
||||
```bash
|
||||
go run ./cmd/crawl -1001234567890 -1009876543210
|
||||
```
|
||||
|
||||
The first crawl asks for the Telegram login code once, then stores the session
|
||||
in Redis. Any later run on any device pointed at the same Redis reuses the
|
||||
Redis config and session and **does not** prompt again.
|
||||
|
||||
Group IDs use the usual marked form: `-100<id>` for supergroups and channels,
|
||||
`-<id>` for basic groups. The account must already be a member of each group.
|
||||
|
||||
## Compare two crawls
|
||||
|
||||
Compare membership changes for one group between two saved crawls with
|
||||
git-diff-style terminal output:
|
||||
|
||||
```bash
|
||||
go run ./cmd/compare <group_id> <time1> <time2>
|
||||
```
|
||||
|
||||
If `time1` and `time2` are omitted, the command compares the latest two crawls
|
||||
for that group:
|
||||
|
||||
```bash
|
||||
go run ./cmd/compare <group_id>
|
||||
```
|
||||
|
||||
If `group_id` is also omitted, the command uses the first group in the stored
|
||||
configuration:
|
||||
|
||||
```bash
|
||||
go run ./cmd/compare
|
||||
```
|
||||
|
||||
## Compare two groups
|
||||
|
||||
Fetch two groups live from Telegram and compare their members by Telegram ID:
|
||||
|
||||
```bash
|
||||
go run ./cmd/compare-groups <group_id_a> <group_id_b>
|
||||
```
|
||||
|
||||
Members in both groups are listed unprefixed, members only in the first group
|
||||
with `-`, and members only in the second group with `+`. No crawl is saved to
|
||||
Redis; the command uses Redis only for the Telegram config and login session,
|
||||
the same way `crawl` does.
|
||||
|
||||
## Export a crawl to CSV
|
||||
|
||||
Export one group's saved crawl from Redis:
|
||||
|
||||
```bash
|
||||
go run ./cmd/export [group_id] [timecrawl]
|
||||
```
|
||||
|
||||
When `timecrawl` is omitted, the latest crawl for the group is exported. When
|
||||
both arguments are omitted, the first group in the stored configuration and its
|
||||
latest crawl are used. An explicit crawl time must use `yyyymmddhhmmss` format:
|
||||
|
||||
```bash
|
||||
go run ./cmd/export -1001234567890 20260724120000
|
||||
```
|
||||
|
||||
CSV files are written to:
|
||||
|
||||
```text
|
||||
output/<group-id>-<yyyymmddhhmmss>.csv
|
||||
```
|
||||
|
||||
Each file contains the columns `id`, `username`, `first_name`, and `last_name`.
|
||||
Formula-like Telegram text is prefixed with an apostrophe for safe spreadsheet
|
||||
opening. On POSIX systems, the `output/` directory is created with `0700`
|
||||
permissions and each CSV file is written with `0600` permissions. The `output/`
|
||||
directory is ignored by Git.
|
||||
|
||||
## Build binaries
|
||||
|
||||
The code is pure Go, so static binaries build without cgo:
|
||||
|
||||
```bash
|
||||
CGO_ENABLED=0 go build -o bin/ ./cmd/...
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `REDIS_URL` | Redis connection string (`rediss://default:<password>@<host>:<port>`) |
|
||||
|
||||
Telegram `api_id`, `api_hash`, `phone`, and `group_ids` are stored in Redis by `configure`.
|
||||
|
||||
## How data is stored
|
||||
|
||||
All data lives in Redis under the `telegram-export` prefix. No key references another, so deleting any key can never corrupt another's state:
|
||||
|
||||
```
|
||||
telegram-export:config -> JSON config:
|
||||
{ api_id, api_hash, phone, group_ids }
|
||||
telegram-export:session-gotd -> gotd session JSON (login)
|
||||
telegram-export:group:<group_id>:<yyyymmddhhmmss> -> one group's export as JSON:
|
||||
{ group_id, title, time,
|
||||
members: [{ id, username, first_name, last_name }] }
|
||||
```
|
||||
|
||||
All groups in one crawl share one timestamp. The login session grants full
|
||||
account access, so restrict Redis access accordingly; deleting
|
||||
`telegram-export:session-gotd` forces a clean re-login.
|
||||
|
||||
### Upgrading from the Python version
|
||||
|
||||
The config and export keys keep the same format, so existing crawls stay
|
||||
readable by `compare` and `export`. The Python version kept a Telethon session
|
||||
under `telegram-export:session`. On first run, the Go version converts it into
|
||||
`telegram-export:session-gotd`, so you do not need a new login code. The old key
|
||||
is only read, never changed, and can be deleted afterwards.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
go vet ./...
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Tests run offline: Redis is faked with miniredis, and Telegram calls are not
|
||||
exercised.
|
||||
@@ -0,0 +1,10 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/commands/cli"
|
||||
comparegroupscmd "github.com/tiennm99/export-telegram-group-members/internal/commands/comparegroups"
|
||||
)
|
||||
|
||||
func main() {
|
||||
cli.Main(comparegroupscmd.Run)
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/commands/cli"
|
||||
comparecmd "github.com/tiennm99/export-telegram-group-members/internal/commands/compare"
|
||||
)
|
||||
|
||||
func main() {
|
||||
cli.Main(comparecmd.Run)
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/commands/cli"
|
||||
configurecmd "github.com/tiennm99/export-telegram-group-members/internal/commands/configure"
|
||||
)
|
||||
|
||||
func main() {
|
||||
cli.Main(configurecmd.Run)
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/commands/cli"
|
||||
crawlcmd "github.com/tiennm99/export-telegram-group-members/internal/commands/crawl"
|
||||
)
|
||||
|
||||
func main() {
|
||||
cli.Main(crawlcmd.Run)
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/commands/cli"
|
||||
exportcmd "github.com/tiennm99/export-telegram-group-members/internal/commands/export"
|
||||
)
|
||||
|
||||
func main() {
|
||||
cli.Main(exportcmd.Run)
|
||||
}
|
||||
-131
@@ -1,131 +0,0 @@
|
||||
import argparse
|
||||
import sys
|
||||
|
||||
RESET = '\033[0m'
|
||||
GREEN = '\033[32m'
|
||||
RED = '\033[31m'
|
||||
CYAN = '\033[36m'
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(
|
||||
description='Compare member changes between two crawls for one group.',
|
||||
)
|
||||
parser.add_argument(
|
||||
'group_id',
|
||||
type=int,
|
||||
nargs='?',
|
||||
help='Telegram group id to compare (defaults to the first configured group)',
|
||||
)
|
||||
parser.add_argument(
|
||||
'times',
|
||||
nargs='*',
|
||||
help='Optional pair: time1 time2. When omitted, latest two crawls are used.',
|
||||
)
|
||||
args = parser.parse_args()
|
||||
if len(args.times) not in (0, 2):
|
||||
parser.error('provide both time1 and time2, or omit both to compare latest two crawls')
|
||||
return args
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
|
||||
# Lazy import: keeps `python compare.py --help` working when REDIS_URL is unset.
|
||||
from common import diff_group_members, get_group_export, list_group_exports
|
||||
|
||||
group_id = args.group_id
|
||||
if group_id is None:
|
||||
from config import load_app_config
|
||||
|
||||
group_id = load_app_config()['group_ids'][0]
|
||||
|
||||
if args.times:
|
||||
before_time, after_time = args.times
|
||||
before_record = get_group_export(group_id, before_time)
|
||||
after_record = get_group_export(group_id, after_time)
|
||||
if before_record is None:
|
||||
print(f'export not found for group {group_id} at {before_time}', file=sys.stderr)
|
||||
return 1
|
||||
if after_record is None:
|
||||
print(f'export not found for group {group_id} at {after_time}', file=sys.stderr)
|
||||
return 1
|
||||
else:
|
||||
exports = list_group_exports(group_id)
|
||||
if len(exports) < 2:
|
||||
print(f'need at least 2 exports for group {group_id}', file=sys.stderr)
|
||||
return 1
|
||||
before_record, after_record = exports[-2:]
|
||||
|
||||
added, removed = diff_group_members(before_record, after_record)
|
||||
print_summary(group_id, before_record, after_record, added, removed)
|
||||
return 0
|
||||
|
||||
|
||||
def print_summary(group_id, before_record, after_record, added, removed):
|
||||
title = after_record.get('title') or before_record.get('title') or ''
|
||||
before_members = before_record.get('members', [])
|
||||
after_members = after_record.get('members', [])
|
||||
before_time = before_record.get('time')
|
||||
after_time = after_record.get('time')
|
||||
label = f'{title} ({group_id})' if title else str(group_id)
|
||||
|
||||
print(f'diff --telegram-group "{label}"')
|
||||
print(color(f'--- crawl/{before_time}', RED))
|
||||
print(color(f'+++ crawl/{after_time}', GREEN))
|
||||
print(color(
|
||||
f'@@ members: {len(before_members)} -> {len(after_members)} '
|
||||
f'(+{len(added)} -{len(removed)}) @@',
|
||||
CYAN,
|
||||
))
|
||||
print()
|
||||
|
||||
widths = column_widths([*removed, *added])
|
||||
for member in removed:
|
||||
print(color(format_member_row('-', member, widths), RED))
|
||||
for member in added:
|
||||
print(color(format_member_row('+', member, widths), GREEN))
|
||||
|
||||
if not added and not removed:
|
||||
print(' no membership changes.')
|
||||
|
||||
|
||||
def color(text, code):
|
||||
if not sys.stdout.isatty():
|
||||
return text
|
||||
return f'{code}{text}{RESET}'
|
||||
|
||||
|
||||
def member_columns(member):
|
||||
username = member.get('username')
|
||||
full_name = ' '.join(
|
||||
part for part in [member.get('first_name'), member.get('last_name')]
|
||||
if part
|
||||
)
|
||||
return [
|
||||
str(member.get('id')),
|
||||
f'@{username}' if username else '',
|
||||
full_name,
|
||||
]
|
||||
|
||||
|
||||
def column_widths(members):
|
||||
widths = [2, 8, 4]
|
||||
for member in members:
|
||||
for index, value in enumerate(member_columns(member)):
|
||||
widths[index] = max(widths[index], len(value))
|
||||
return widths
|
||||
|
||||
|
||||
def format_member_row(prefix, member, widths):
|
||||
member_id, username, full_name = member_columns(member)
|
||||
return (
|
||||
f'{prefix} '
|
||||
f'{member_id:<{widths[0]}} '
|
||||
f'{username:<{widths[1]}} '
|
||||
f'{full_name:<{widths[2]}}'
|
||||
).rstrip()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
@@ -1,91 +0,0 @@
|
||||
import argparse
|
||||
import sys
|
||||
|
||||
from compare import CYAN, GREEN, RED, color, column_widths, format_member_row
|
||||
|
||||
|
||||
def parse_args(argv=None):
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
'Fetch two Telegram groups live and compare their members. '
|
||||
'Nothing is saved to Redis.'
|
||||
),
|
||||
)
|
||||
parser.add_argument('group_a', type=int, help='first Telegram group id')
|
||||
parser.add_argument('group_b', type=int, help='second Telegram group id')
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
args = parse_args(argv)
|
||||
|
||||
# Lazy imports: keep `--help` working when REDIS_URL is unset. Redis still
|
||||
# supplies the Telegram config and login session; no export is written.
|
||||
from common import member_dict
|
||||
from config import load_app_config
|
||||
from crawl import fetch_group, login
|
||||
|
||||
client = login(load_app_config())
|
||||
|
||||
groups = []
|
||||
for group_id in (args.group_a, args.group_b):
|
||||
try:
|
||||
entity, members = fetch_group(client, group_id, 'fetching')
|
||||
except Exception as e:
|
||||
print(f'error accessing group {group_id}: {e}', file=sys.stderr)
|
||||
return 1
|
||||
if entity is None:
|
||||
print(f'{group_id} is not a group.', file=sys.stderr)
|
||||
return 1
|
||||
groups.append({
|
||||
'group_id': group_id,
|
||||
'title': entity.title,
|
||||
'members': [member_dict(m) for m in members],
|
||||
})
|
||||
|
||||
group_a, group_b = groups
|
||||
only_a, only_b, both = split_members(group_a, group_b)
|
||||
print()
|
||||
print_summary(group_a, group_b, only_a, only_b, both)
|
||||
return 0
|
||||
|
||||
|
||||
def split_members(group_a, group_b):
|
||||
"""Return (only in A, only in B, in both) by Telegram member id."""
|
||||
a_members = {m['id']: m for m in group_a['members']}
|
||||
b_members = {m['id']: m for m in group_b['members']}
|
||||
only_a = [a_members[i] for i in sorted(a_members.keys() - b_members.keys())]
|
||||
only_b = [b_members[i] for i in sorted(b_members.keys() - a_members.keys())]
|
||||
both = [b_members[i] for i in sorted(a_members.keys() & b_members.keys())]
|
||||
return only_a, only_b, both
|
||||
|
||||
|
||||
def group_label(group):
|
||||
return f'{group["title"]} ({group["group_id"]})'
|
||||
|
||||
|
||||
def print_summary(group_a, group_b, only_a, only_b, both):
|
||||
print(f'diff --telegram-groups "{group_label(group_a)}" "{group_label(group_b)}"')
|
||||
print(color(f'--- {group_label(group_a)}', RED))
|
||||
print(color(f'+++ {group_label(group_b)}', GREEN))
|
||||
print(color(
|
||||
f'@@ members: {len(group_a["members"])} vs {len(group_b["members"])} '
|
||||
f'(both {len(both)}, -{len(only_a)} only first, +{len(only_b)} only second) @@',
|
||||
CYAN,
|
||||
))
|
||||
print()
|
||||
|
||||
widths = column_widths([*only_a, *only_b, *both])
|
||||
for member in both:
|
||||
print(format_member_row(' ', member, widths))
|
||||
for member in only_a:
|
||||
print(color(format_member_row('-', member, widths), RED))
|
||||
for member in only_b:
|
||||
print(color(format_member_row('+', member, widths), GREEN))
|
||||
|
||||
if not group_a['members'] and not group_b['members']:
|
||||
print(' both groups have no visible members.')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
@@ -1,76 +0,0 @@
|
||||
import argparse
|
||||
import getpass
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(
|
||||
description='Store Telegram crawl config in Redis.',
|
||||
)
|
||||
parser.add_argument('--api-id', type=int, help='Telegram API ID')
|
||||
parser.add_argument('--api-hash', help='Telegram API hash')
|
||||
parser.add_argument('--phone', help='Telegram account phone')
|
||||
parser.add_argument(
|
||||
'--groups',
|
||||
help='Comma-separated Telegram group IDs to crawl',
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def prompt_required(label):
|
||||
value = input(f'{label}: ').strip()
|
||||
if not value:
|
||||
raise SystemExit(f'{label.lower()} is required')
|
||||
return value
|
||||
|
||||
|
||||
def prompt_api_id(value):
|
||||
if value is not None:
|
||||
return value
|
||||
try:
|
||||
return int(prompt_required('API ID'))
|
||||
except ValueError as exc:
|
||||
raise SystemExit('api id must be an integer') from exc
|
||||
|
||||
|
||||
def prompt_api_hash(value):
|
||||
api_hash = value or getpass.getpass('API hash: ')
|
||||
if not api_hash:
|
||||
raise SystemExit('api hash is required')
|
||||
return api_hash
|
||||
|
||||
|
||||
def prompt_phone(value):
|
||||
return value or prompt_required('Phone')
|
||||
|
||||
|
||||
def prompt_groups(value):
|
||||
return parse_group_ids(value or prompt_required('Group IDs'))
|
||||
|
||||
|
||||
def parse_group_ids(value):
|
||||
group_ids = [part.strip() for part in value.split(',') if part.strip()]
|
||||
if not group_ids:
|
||||
raise SystemExit('at least one group id is required')
|
||||
try:
|
||||
return [int(group_id) for group_id in group_ids]
|
||||
except ValueError as exc:
|
||||
raise SystemExit('group ids must be comma-separated integers') from exc
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
api_id = prompt_api_id(args.api_id)
|
||||
api_hash = prompt_api_hash(args.api_hash)
|
||||
phone = prompt_phone(args.phone)
|
||||
group_ids = prompt_groups(args.groups)
|
||||
|
||||
# Lazy import: keeps `python configure.py --help` working when REDIS_URL is unset.
|
||||
from config import save_app_config
|
||||
|
||||
config = save_app_config(api_id, api_hash, phone, group_ids)
|
||||
print(f'saved Redis config for {config["phone"]}: {len(group_ids)} group(s).')
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
@@ -1,87 +0,0 @@
|
||||
import argparse
|
||||
import getpass
|
||||
|
||||
from telethon.errors import SessionPasswordNeededError
|
||||
from telethon.sessions import StringSession
|
||||
from telethon.sync import TelegramClient
|
||||
from telethon.tl.types import Chat, Channel
|
||||
|
||||
from common import new_run_time, save_group_export
|
||||
from config import load_app_config
|
||||
from session_store import load_session, save_session
|
||||
|
||||
|
||||
def parse_args(argv=None):
|
||||
parser = argparse.ArgumentParser(
|
||||
description='Export members from Telegram groups.',
|
||||
)
|
||||
parser.add_argument(
|
||||
'group_ids',
|
||||
nargs='*',
|
||||
type=int,
|
||||
metavar='group_id',
|
||||
help='Telegram group ID to crawl; defaults to the configured group IDs',
|
||||
)
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def login(app_config):
|
||||
"""Connect to Telegram, signing in only when the stored session is missing."""
|
||||
# Session loads from Redis: a saved string means no re-login on any device.
|
||||
client = TelegramClient(
|
||||
StringSession(load_session()),
|
||||
app_config['api_id'],
|
||||
app_config['api_hash'],
|
||||
)
|
||||
|
||||
client.connect()
|
||||
if not client.is_user_authorized():
|
||||
client.send_code_request(app_config['phone'])
|
||||
try:
|
||||
client.sign_in(code=input('Enter code: '))
|
||||
except SessionPasswordNeededError:
|
||||
client.sign_in(password=getpass.getpass())
|
||||
|
||||
save_session(client.session.save())
|
||||
return client
|
||||
|
||||
|
||||
def fetch_group(client, group_id, action):
|
||||
"""Return (entity, members sorted by id), or (None, []) for a non-group."""
|
||||
entity = client.get_entity(group_id)
|
||||
if not isinstance(entity, (Chat, Channel)):
|
||||
return None, []
|
||||
print(f'{action} {entity.title} (ID: {group_id})')
|
||||
members = client.get_participants(entity)
|
||||
members.sort(key=lambda x: x.id)
|
||||
return entity, members
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
args = parse_args(argv)
|
||||
app_config = load_app_config()
|
||||
# Command-line IDs override the configured list for one-off crawls.
|
||||
group_ids = args.group_ids or app_config['group_ids']
|
||||
|
||||
client = login(app_config)
|
||||
|
||||
run_time = new_run_time()
|
||||
saved = 0
|
||||
for group_id in group_ids:
|
||||
try:
|
||||
entity, members = fetch_group(client, group_id, 'exporting')
|
||||
if entity is None:
|
||||
print(f'skip {group_id} because it is not a group.')
|
||||
continue
|
||||
save_group_export(group_id, entity.title, members, run_time)
|
||||
saved += 1
|
||||
print(f'export {entity.title} done.')
|
||||
except Exception as e:
|
||||
print(f'error accessing group {group_id}: {e}')
|
||||
|
||||
print(f'saved run {run_time}: {saved} group(s) to Redis.')
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
@@ -1,169 +0,0 @@
|
||||
import argparse
|
||||
import csv
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
OUTPUT_DIR = Path('output')
|
||||
CSV_FIELDS = ['id', 'username', 'first_name', 'last_name']
|
||||
FORMULA_PREFIXES = ('=', '+', '-', '@', '\t', '\r')
|
||||
|
||||
|
||||
def parse_args(argv=None):
|
||||
parser = argparse.ArgumentParser(
|
||||
description='Export a Redis-backed Telegram group crawl to CSV.',
|
||||
)
|
||||
parser.add_argument(
|
||||
'group_id',
|
||||
type=int,
|
||||
nargs='?',
|
||||
help='Telegram group ID (defaults to the first configured group)',
|
||||
)
|
||||
parser.add_argument(
|
||||
'timecrawl',
|
||||
nargs='?',
|
||||
type=parse_run_time,
|
||||
help='Crawl time in yyyymmddhhmmss format (defaults to latest)',
|
||||
)
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def parse_run_time(value):
|
||||
if not is_valid_run_time(value):
|
||||
if len(value) != 14 or not value.isdigit():
|
||||
raise argparse.ArgumentTypeError(
|
||||
'timecrawl must use yyyymmddhhmmss format',
|
||||
)
|
||||
raise argparse.ArgumentTypeError(
|
||||
'timecrawl must be a valid date and time',
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def is_valid_run_time(value):
|
||||
if not isinstance(value, str) or len(value) != 14 or not value.isdigit():
|
||||
return False
|
||||
try:
|
||||
datetime.strptime(value, '%Y%m%d%H%M%S')
|
||||
except ValueError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
args = parse_args(argv)
|
||||
|
||||
group_id = args.group_id
|
||||
try:
|
||||
# Lazy imports keep `python export.py --help` working without REDIS_URL.
|
||||
from common import get_group_export, latest_group_export_time
|
||||
|
||||
if group_id is None:
|
||||
from config import load_app_config
|
||||
|
||||
group_id = load_app_config()['group_ids'][0]
|
||||
|
||||
run_time = args.timecrawl
|
||||
if run_time is None:
|
||||
run_time = latest_group_export_time(group_id)
|
||||
if run_time is None:
|
||||
print(f'no exports found for group {group_id}', file=sys.stderr)
|
||||
return 1
|
||||
record = get_group_export(group_id, run_time)
|
||||
except Exception as exc:
|
||||
print(f'could not read export configuration or history: {exc}', file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if record is None:
|
||||
print(
|
||||
f'export not found or invalid for group {group_id} at {run_time}',
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
try:
|
||||
members = validate_record(record, group_id, run_time)
|
||||
output_path = write_csv(group_id, run_time, members)
|
||||
except (OSError, ValueError, csv.Error) as exc:
|
||||
print(f'could not export group {group_id} at {run_time}: {exc}', file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print(f'exported {len(members)} members to {output_path}')
|
||||
return 0
|
||||
|
||||
|
||||
def validate_record(record, expected_group_id, expected_time):
|
||||
if not is_valid_run_time(expected_time):
|
||||
raise ValueError('Redis export key has an invalid crawl time')
|
||||
if not isinstance(record, dict):
|
||||
raise ValueError('Redis export is not an object')
|
||||
if record.get('group_id') != expected_group_id:
|
||||
raise ValueError('Redis export group ID does not match its key')
|
||||
if record.get('time') != expected_time:
|
||||
raise ValueError('Redis export time does not match its key')
|
||||
|
||||
members = record.get('members')
|
||||
if not isinstance(members, list):
|
||||
raise ValueError('Redis export members must be a list')
|
||||
|
||||
validated = []
|
||||
for index, member in enumerate(members):
|
||||
if not isinstance(member, dict):
|
||||
raise ValueError(f'member {index} is not an object')
|
||||
member_id = member.get('id')
|
||||
if not isinstance(member_id, int) or isinstance(member_id, bool):
|
||||
raise ValueError(f'member {index} has an invalid ID')
|
||||
|
||||
row = {'id': member_id}
|
||||
for field in CSV_FIELDS[1:]:
|
||||
value = member.get(field)
|
||||
if value is not None and not isinstance(value, str):
|
||||
raise ValueError(f'member {index} has an invalid {field}')
|
||||
row[field] = escape_formula(value)
|
||||
validated.append(row)
|
||||
return validated
|
||||
|
||||
|
||||
def escape_formula(value):
|
||||
if value and (
|
||||
value[0] in ('\t', '\r', '\n')
|
||||
or value.lstrip().startswith(FORMULA_PREFIXES)
|
||||
):
|
||||
return f"'{value}"
|
||||
return value
|
||||
|
||||
|
||||
def write_csv(group_id, run_time, members, output_dir=None):
|
||||
output_dir = output_dir or OUTPUT_DIR
|
||||
output_path = output_dir / f'{group_id}-{run_time}.csv'
|
||||
output_dir.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
output_dir.chmod(0o700)
|
||||
|
||||
temp_path = None
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode='w',
|
||||
encoding='utf-8',
|
||||
newline='',
|
||||
dir=output_dir,
|
||||
prefix=f'.{output_path.name}.',
|
||||
delete=False,
|
||||
) as csv_file:
|
||||
temp_path = Path(csv_file.name)
|
||||
writer = csv.DictWriter(csv_file, fieldnames=CSV_FIELDS)
|
||||
writer.writeheader()
|
||||
writer.writerows(members)
|
||||
os.chmod(temp_path, 0o600)
|
||||
os.replace(temp_path, output_path)
|
||||
except Exception:
|
||||
if temp_path is not None:
|
||||
temp_path.unlink(missing_ok=True)
|
||||
raise
|
||||
|
||||
return output_path
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
@@ -1,63 +1,54 @@
|
||||
module github.com/tiennm99/export-telegram-group-members
|
||||
|
||||
go 1.24.0
|
||||
|
||||
toolchain go1.24.10
|
||||
go 1.27.1
|
||||
|
||||
require (
|
||||
github.com/celestix/gotgproto v1.0.0-beta22
|
||||
github.com/alicebob/miniredis/v2 v2.40.0
|
||||
github.com/gotd/td v0.162.0
|
||||
github.com/joho/godotenv v1.5.1
|
||||
gorm.io/driver/sqlite v1.6.0
|
||||
github.com/mattn/go-isatty v0.0.24
|
||||
github.com/redis/go-redis/v9 v9.23.0
|
||||
golang.org/x/term v0.47.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/AnimeKaizoku/cacher v1.0.3-0.20250508132714-ddc7471efeef // indirect
|
||||
github.com/andybalholm/brotli v1.2.1 // indirect
|
||||
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
|
||||
github.com/coder/websocket v1.8.14 // indirect
|
||||
github.com/dlclark/regexp2 v1.11.5 // indirect
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/fatih/color v1.18.0 // indirect
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
github.com/coder/websocket v1.8.15 // indirect
|
||||
github.com/dlclark/regexp2 v1.12.0 // indirect
|
||||
github.com/fatih/color v1.19.0 // indirect
|
||||
github.com/ghodss/yaml v1.0.0 // indirect
|
||||
github.com/glebarez/go-sqlite v1.22.0 // indirect
|
||||
github.com/glebarez/sqlite v1.11.0 // indirect
|
||||
github.com/go-faster/errors v0.7.1 // indirect
|
||||
github.com/go-faster/jx v1.1.0 // indirect
|
||||
github.com/go-faster/errors v0.8.0 // indirect
|
||||
github.com/go-faster/jx v1.2.0 // indirect
|
||||
github.com/go-faster/xor v1.0.0 // indirect
|
||||
github.com/go-faster/yaml v0.4.6 // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/gotd/ige v0.2.2 // indirect
|
||||
github.com/gotd/ige v0.3.0 // indirect
|
||||
github.com/gotd/log v0.1.0 // indirect
|
||||
github.com/gotd/neo v0.1.5 // indirect
|
||||
github.com/gotd/td v0.132.0 // indirect
|
||||
github.com/jinzhu/inflection v1.0.0 // indirect
|
||||
github.com/jinzhu/now v1.1.5 // indirect
|
||||
github.com/klauspost/compress v1.18.0 // indirect
|
||||
github.com/mattn/go-colorable v0.1.14 // indirect
|
||||
github.com/mattn/go-isatty v0.0.20 // indirect
|
||||
github.com/mattn/go-sqlite3 v1.14.22 // indirect
|
||||
github.com/ncruces/go-strftime v0.1.9 // indirect
|
||||
github.com/ogen-go/ogen v1.15.2 // indirect
|
||||
github.com/pkg/errors v0.9.1 // indirect
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
|
||||
github.com/segmentio/asm v1.2.0 // indirect
|
||||
go.opentelemetry.io/otel v1.38.0 // indirect
|
||||
go.opentelemetry.io/otel/metric v1.38.0 // indirect
|
||||
go.opentelemetry.io/otel/trace v1.38.0 // indirect
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
github.com/klauspost/compress v1.19.1 // indirect
|
||||
github.com/mattn/go-colorable v0.1.15 // indirect
|
||||
github.com/ogen-go/ogen v1.23.0 // indirect
|
||||
github.com/refraction-networking/utls v1.8.2 // indirect
|
||||
github.com/segmentio/asm v1.2.1 // indirect
|
||||
github.com/shopspring/decimal v1.4.0 // indirect
|
||||
github.com/yuin/goldmark v1.8.5 // indirect
|
||||
github.com/yuin/gopher-lua v1.1.1 // indirect
|
||||
go.opentelemetry.io/otel v1.44.0 // indirect
|
||||
go.opentelemetry.io/otel/metric v1.44.0 // indirect
|
||||
go.opentelemetry.io/otel/trace v1.44.0 // indirect
|
||||
go.uber.org/atomic v1.12.0 // indirect
|
||||
go.uber.org/multierr v1.11.0 // indirect
|
||||
go.uber.org/zap v1.27.0 // indirect
|
||||
golang.org/x/crypto v0.43.0 // indirect
|
||||
golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8 // indirect
|
||||
golang.org/x/mod v0.29.0 // indirect
|
||||
golang.org/x/net v0.46.0 // indirect
|
||||
golang.org/x/sync v0.17.0 // indirect
|
||||
golang.org/x/sys v0.37.0 // indirect
|
||||
golang.org/x/text v0.30.0 // indirect
|
||||
golang.org/x/tools v0.38.0 // indirect
|
||||
go.uber.org/zap v1.28.0 // indirect
|
||||
golang.org/x/crypto v0.54.0 // indirect
|
||||
golang.org/x/exp v0.0.0-20230725093048-515e97ebf090 // indirect
|
||||
golang.org/x/mod v0.38.0 // indirect
|
||||
golang.org/x/net v0.57.0 // indirect
|
||||
golang.org/x/sync v0.22.0 // indirect
|
||||
golang.org/x/sys v0.49.0 // indirect
|
||||
golang.org/x/text v0.40.0 // indirect
|
||||
golang.org/x/tools v0.48.0 // indirect
|
||||
gopkg.in/yaml.v2 v2.4.0 // indirect
|
||||
gorm.io/gorm v1.30.0 // indirect
|
||||
modernc.org/libc v1.61.8 // indirect
|
||||
modernc.org/mathutil v1.7.1 // indirect
|
||||
modernc.org/memory v1.8.2 // indirect
|
||||
modernc.org/sqlite v1.34.4 // indirect
|
||||
rsc.io/qr v0.2.0 // indirect
|
||||
)
|
||||
@@ -1,30 +1,29 @@
|
||||
github.com/AnimeKaizoku/cacher v1.0.3-0.20250508132714-ddc7471efeef h1:y8llZexgesBazUg/zdIlrZKHPFJ2zk9YxlLAJ5x4DpQ=
|
||||
github.com/AnimeKaizoku/cacher v1.0.3-0.20250508132714-ddc7471efeef/go.mod h1:jw0de/b0K6W7Y3T9rHCMGVKUf6oG7hENNcssxYcZTCc=
|
||||
github.com/celestix/gotgproto v1.0.0-beta22 h1:Iu78cFA08nV8+flmxKs9CJ3W73+HG30fx0nLOs5A6fI=
|
||||
github.com/celestix/gotgproto v1.0.0-beta22/go.mod h1:JYC9Js/5KLUhFR5M2RslQi2DFAcF7EdrgJMXo0YrzGQ=
|
||||
github.com/alicebob/miniredis/v2 v2.40.0 h1:6IgTQbvm2JiMYZ8yur/acNJzVjtSOWCelxXbk/LUBXc=
|
||||
github.com/alicebob/miniredis/v2 v2.40.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM=
|
||||
github.com/andybalholm/brotli v1.2.1 h1:R+f5xP285VArJDRgowrfb9DqL18yVK0gKAW/F+eTWro=
|
||||
github.com/andybalholm/brotli v1.2.1/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
|
||||
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
|
||||
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
|
||||
github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
|
||||
github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
|
||||
github.com/coder/websocket v1.8.14 h1:9L0p0iKiNOibykf283eHkKUHHrpG7f65OE3BhhO7v9g=
|
||||
github.com/coder/websocket v1.8.14/go.mod h1:NX3SzP+inril6yawo5CQXx8+fk145lPDC6pumgx0mVg=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/coder/websocket v1.8.15 h1:6B2JPeOGlpff2Uz6vOEH1Vzpi0iUz20A+lPVhPHtNUA=
|
||||
github.com/coder/websocket v1.8.15/go.mod h1:NX3SzP+inril6yawo5CQXx8+fk145lPDC6pumgx0mVg=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dlclark/regexp2 v1.11.5 h1:Q/sSnsKerHeCkc/jSTNq1oCm7KiVgUMZRDUoRu0JQZQ=
|
||||
github.com/dlclark/regexp2 v1.11.5/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
|
||||
github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
|
||||
github.com/dlclark/regexp2 v1.12.0 h1:0j4c5qQmnC6XOWNjP3PIXURXN2gWx76rd3KvgdPkCz8=
|
||||
github.com/dlclark/regexp2 v1.12.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||
github.com/fatih/color v1.19.0 h1:Zp3PiM21/9Ld6FzSKyL5c/BULoe/ONr9KlbYVOfG8+w=
|
||||
github.com/fatih/color v1.19.0/go.mod h1:zNk67I0ZUT1bEGsSGyCZYZNrHuTkJJB+r6Q9VuMi0LE=
|
||||
github.com/ghodss/yaml v1.0.0 h1:wQHKEahhL6wmXdzwWG11gIVCkOv05bNOh+Rxn0yngAk=
|
||||
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
|
||||
github.com/glebarez/go-sqlite v1.22.0 h1:uAcMJhaA6r3LHMTFgP0SifzgXg46yJkgxqyuyec+ruQ=
|
||||
github.com/glebarez/go-sqlite v1.22.0/go.mod h1:PlBIdHe0+aUEFn+r2/uthrWq4FxbzugL0L8Li6yQJbc=
|
||||
github.com/glebarez/sqlite v1.11.0 h1:wSG0irqzP6VurnMEpFGer5Li19RpIRi2qvQz++w0GMw=
|
||||
github.com/glebarez/sqlite v1.11.0/go.mod h1:h8/o8j5wiAsqSPoWELDUdJXhjAhsVliSn7bWZjOhrgQ=
|
||||
github.com/go-faster/errors v0.7.1 h1:MkJTnDoEdi9pDabt1dpWf7AA8/BaSYZqibYyhZ20AYg=
|
||||
github.com/go-faster/errors v0.7.1/go.mod h1:5ySTjWFiphBs07IKuiL69nxdfd5+fzh1u7FPGZP2quo=
|
||||
github.com/go-faster/jx v1.1.0 h1:ZsW3wD+snOdmTDy9eIVgQdjUpXRRV4rqW8NS3t+20bg=
|
||||
github.com/go-faster/jx v1.1.0/go.mod h1:vKDNikrKoyUmpzaJ0OkIkRQClNHFX/nF3dnTJZb3skg=
|
||||
github.com/go-faster/xor v0.3.0/go.mod h1:x5CaDY9UKErKzqfRfFZdfu+OSTfoZny3w5Ak7UxcipQ=
|
||||
github.com/go-faster/errors v0.8.0 h1:9T9eJrM+72dFk7n4DfhuaDDe6cyuFCSW2oNUkN77Yqc=
|
||||
github.com/go-faster/errors v0.8.0/go.mod h1:5ySTjWFiphBs07IKuiL69nxdfd5+fzh1u7FPGZP2quo=
|
||||
github.com/go-faster/jx v1.2.0 h1:T2YHJPrFaYu21fJtUxC9GzmluKu8rVIFDwwGBKTDseI=
|
||||
github.com/go-faster/jx v1.2.0/go.mod h1:UWLOVDmMG597a5tBFPLIWJdUxz5/2emOpfsj9Neg0PE=
|
||||
github.com/go-faster/xor v1.0.0 h1:2o8vTOgErSGHP3/7XwA5ib1FTtUsNtwCoLLBjl31X38=
|
||||
github.com/go-faster/xor v1.0.0/go.mod h1:x5CaDY9UKErKzqfRfFZdfu+OSTfoZny3w5Ak7UxcipQ=
|
||||
github.com/go-faster/yaml v0.4.6 h1:lOK/EhI04gCpPgPhgt0bChS6bvw7G3WwI8xxVe0sw9I=
|
||||
@@ -35,117 +34,94 @@ github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
|
||||
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||
github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd h1:gbpYu9NMq8jhDVbvlGkMFWCjLFlqqEZjEmObmhUy6Vo=
|
||||
github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd/go.mod h1:kf6iHlnVGwgKolg33glAes7Yg/8iWP8ukqeldJSO7jw=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/gotd/ige v0.2.2 h1:XQ9dJZwBfDnOGSTxKXBGP4gMud3Qku2ekScRjDWWfEk=
|
||||
github.com/gotd/ige v0.2.2/go.mod h1:tuCRb+Y5Y3eNTo3ypIfNpQ4MFjrnONiL2jN2AKZXmb0=
|
||||
github.com/gotd/ige v0.3.0 h1:4f6LEHWsVDLBG0bT9wWG2/9TZb5aWm265G8ZlTXmRRU=
|
||||
github.com/gotd/ige v0.3.0/go.mod h1:FE9bTaQtvfArizAcZuI4sS6gXaEUBmixdUufVHoCKac=
|
||||
github.com/gotd/log v0.1.0 h1:4LJUEvafD1xtBwx2QkrlzFnRgbYXTlWqJPDi8BvrLbU=
|
||||
github.com/gotd/log v0.1.0/go.mod h1:5ilhdu1Ux0QvDY/FF3Ojfw24Ws3SlCtyLwOpXy8KYXs=
|
||||
github.com/gotd/log/logzap v0.1.1 h1:O6l7d8HUbODe+UMcrM47eXYDwdJ6RNmpQejLjrlcEIQ=
|
||||
github.com/gotd/log/logzap v0.1.1/go.mod h1:5ObZkITbfhbsBOLzBkzmMk9QxXc0eNQpimau7zRL+Y8=
|
||||
github.com/gotd/neo v0.1.5 h1:oj0iQfMbGClP8xI59x7fE/uHoTJD7NZH9oV1WNuPukQ=
|
||||
github.com/gotd/neo v0.1.5/go.mod h1:9A2a4bn9zL6FADufBdt7tZt+WMhvZoc5gWXihOPoiBQ=
|
||||
github.com/gotd/td v0.132.0 h1:Iqm3S2b+8kDgA9237IDXRxj7sryUpvy+4Cr50/0tpx4=
|
||||
github.com/gotd/td v0.132.0/go.mod h1:4CDGYS+rDtOqotRheGaF9MS5g6jaUewvSXqBNJnx8SQ=
|
||||
github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E=
|
||||
github.com/jinzhu/inflection v1.0.0/go.mod h1:h+uFLlag+Qp1Va5pdKtLDYj+kHp5pxUVkryuEj+Srlc=
|
||||
github.com/jinzhu/now v1.1.5 h1:/o9tlHleP7gOFmsnYNz3RGnqzefHA47wQpKrrdTIwXQ=
|
||||
github.com/jinzhu/now v1.1.5/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8=
|
||||
github.com/gotd/td v0.162.0 h1:3pUzBsEbb0I5QquvfSkXihkVBXI+QTtdd/upijjUbkw=
|
||||
github.com/gotd/td v0.162.0/go.mod h1:ZsGbErIos7XHF7HL0VlsjL6mmX4b1JpinHfDb7gOdvs=
|
||||
github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
|
||||
github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
|
||||
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
|
||||
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
|
||||
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
|
||||
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||
github.com/klauspost/cpuid/v2 v2.4.0 h1:S6Hrbc7+ywsr0r+RLapfGBHfyefhCTwEh3A0tV913Dw=
|
||||
github.com/klauspost/cpuid/v2 v2.4.0/go.mod h1:19jmZ9mjzoF//ddRSUsv0zfBTJWh3QJh9FNxZTMrGxU=
|
||||
github.com/kr/pretty v0.2.1 h1:Fmg33tUaq4/8ym9TJN1x7sLJnHVwhP33CNkpYV/7rwI=
|
||||
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
|
||||
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
||||
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
||||
github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE=
|
||||
github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
|
||||
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
|
||||
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
|
||||
github.com/mattn/go-sqlite3 v1.14.22 h1:2gZY6PC6kBnID23Tichd1K+Z0oS6nE/XwU+Vz/5o4kU=
|
||||
github.com/mattn/go-sqlite3 v1.14.22/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
|
||||
github.com/ncruces/go-strftime v0.1.9 h1:bY0MQC28UADQmHmaF5dgpLmImcShSi2kHU9XLdhx/f4=
|
||||
github.com/ncruces/go-strftime v0.1.9/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
|
||||
github.com/ogen-go/ogen v1.15.2 h1:Hy5XNcDgWur758Kf0+DTQFN8cyBOs58EjDD3NMqih54=
|
||||
github.com/ogen-go/ogen v1.15.2/go.mod h1:bS+BP2cV7+IGjOM24znBmh+PrpZvYFXA7o3BNF4Hj2E=
|
||||
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
|
||||
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
|
||||
github.com/segmentio/asm v1.2.0 h1:9BQrFxC+YOHJlTlHGkTrFWf59nbL3XnCoFLTwDCI7ys=
|
||||
github.com/segmentio/asm v1.2.0/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
|
||||
go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
|
||||
go.opentelemetry.io/otel v1.38.0 h1:RkfdswUDRimDg0m2Az18RKOsnI8UDzppJAtj01/Ymk8=
|
||||
go.opentelemetry.io/otel v1.38.0/go.mod h1:zcmtmQ1+YmQM9wrNsTGV/q/uyusom3P8RxwExxkZhjM=
|
||||
go.opentelemetry.io/otel/metric v1.38.0 h1:Kl6lzIYGAh5M159u9NgiRkmoMKjvbsKtYRwgfrA6WpA=
|
||||
go.opentelemetry.io/otel/metric v1.38.0/go.mod h1:kB5n/QoRM8YwmUahxvI3bO34eVtQf2i4utNVLr9gEmI=
|
||||
go.opentelemetry.io/otel/trace v1.38.0 h1:Fxk5bKrDZJUH+AMyyIXGcFAPah0oRcT+LuNtJrmcNLE=
|
||||
go.opentelemetry.io/otel/trace v1.38.0/go.mod h1:j1P9ivuFsTceSWe1oY+EeW3sc+Pp42sO++GHkg4wwhs=
|
||||
go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
|
||||
go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
|
||||
github.com/mattn/go-colorable v0.1.15 h1:+u9SLTRGnXv73cEsnsmoZBom+dMU88B2M0aDcWy0/jY=
|
||||
github.com/mattn/go-colorable v0.1.15/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
|
||||
github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI=
|
||||
github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A=
|
||||
github.com/ogen-go/ogen v1.23.0 h1:QaWeKm2KZ2zy7NkqqO1Vdl5idNqlG+svxdgwVAX+zbo=
|
||||
github.com/ogen-go/ogen v1.23.0/go.mod h1:bwwvC3AmCV+LrL5lazyQwwof90402mdcSyI0FOzzpfM=
|
||||
github.com/redis/go-redis/v9 v9.23.0 h1:/TTVdJa8BRNsybwwGGnmiYktHNlUXtbOQxlWpHBPL3s=
|
||||
github.com/redis/go-redis/v9 v9.23.0/go.mod h1:EWP2UUk+XDaI1s5nQB5mablZN6vccXlmBQTyfnF87D0=
|
||||
github.com/refraction-networking/utls v1.8.2 h1:j4Q1gJj0xngdeH+Ox/qND11aEfhpgoEvV+S9iJ2IdQo=
|
||||
github.com/refraction-networking/utls v1.8.2/go.mod h1:jkSOEkLqn+S/jtpEHPOsVv/4V4EVnelwbMQl4vCWXAM=
|
||||
github.com/segmentio/asm v1.2.1 h1:DTNbBqs57ioxAD4PrArqftgypG4/qNpXoJx8TVXxPR0=
|
||||
github.com/segmentio/asm v1.2.1/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs=
|
||||
github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
|
||||
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
|
||||
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
|
||||
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
|
||||
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
|
||||
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
|
||||
github.com/yuin/goldmark v1.8.5 h1:r6N5afV5qj/5S4UTch8agZHJ8UxNCMwX7WjkkJam2NA=
|
||||
github.com/yuin/goldmark v1.8.5/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
|
||||
github.com/yuin/gopher-lua v1.1.1 h1:kYKnWBjvbNP4XLT3+bPEwAXJx262OhaHDWDVOPjL46M=
|
||||
github.com/yuin/gopher-lua v1.1.1/go.mod h1:GBR0iDaNXjAgGg9zfCvksxSRnQx76gclCIb7kdAd1Pw=
|
||||
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
|
||||
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
|
||||
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
|
||||
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
|
||||
go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=
|
||||
go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=
|
||||
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
|
||||
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
|
||||
go.uber.org/atomic v1.12.0 h1:BvcXdFKuviU4fTL/f+SxdQ5qJX/Jix8pAkgdUcb3XOE=
|
||||
go.uber.org/atomic v1.12.0/go.mod h1:I6c4cg+6HCxRjfjSsYtApoFILnpc0CGUdGkXVqbYVNk=
|
||||
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
|
||||
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
|
||||
go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
|
||||
go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
||||
go.uber.org/zap v1.27.0 h1:aJMhYGrd5QSmlpLMr2MftRKl7t8J8PTZPA732ud/XR8=
|
||||
go.uber.org/zap v1.27.0/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E=
|
||||
golang.org/x/crypto v0.43.0 h1:dduJYIi3A3KOfdGOHX8AVZ/jGiyPa3IbBozJ5kNuE04=
|
||||
golang.org/x/crypto v0.43.0/go.mod h1:BFbav4mRNlXJL4wNeejLpWxB7wMbc79PdRGhWKncxR0=
|
||||
golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8 h1:yqrTHse8TCMW1M1ZCP+VAR/l0kKxwaAIqN/il7x4voA=
|
||||
golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8/go.mod h1:tujkw807nyEEAamNbDrEGzRav+ilXA7PCRAd6xsmwiU=
|
||||
golang.org/x/mod v0.29.0 h1:HV8lRxZC4l2cr3Zq1LvtOsi/ThTgWnUk/y64QSs8GwA=
|
||||
golang.org/x/mod v0.29.0/go.mod h1:NyhrlYXJ2H4eJiRy/WDBO6HMqZQ6q9nk4JzS3NuCK+w=
|
||||
golang.org/x/net v0.46.0 h1:giFlY12I07fugqwPuWJi68oOnpfqFnJIJzaIIm2JVV4=
|
||||
golang.org/x/net v0.46.0/go.mod h1:Q9BGdFy1y4nkUwiLvT5qtyhAnEHgnQ/zd8PfU6nc210=
|
||||
go.uber.org/zap v1.28.0 h1:IZzaP1Fv73/T/pBMLk4VutPl36uNC+OSUh3JLG3FIjo=
|
||||
go.uber.org/zap v1.28.0/go.mod h1:rDLpOi171uODNm/mxFcuYWxDsqWSAVkFdX4XojSKg/Q=
|
||||
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
||||
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
|
||||
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
|
||||
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
|
||||
golang.org/x/exp v0.0.0-20230725093048-515e97ebf090 h1:Di6/M8l0O2lCLc6VVRWhgCiApHV8MnQurBnFSHsQtNY=
|
||||
golang.org/x/exp v0.0.0-20230725093048-515e97ebf090/go.mod h1:FXUEEKJgO7OQYeo8N01OfiKP8RXMtf6e8aTskBGqWdc=
|
||||
golang.org/x/mod v0.38.0 h1:MECBjubtXD7yj4HrhIUcywNaGeNVUdfVnxmPajOk4yk=
|
||||
golang.org/x/mod v0.38.0/go.mod h1:V6Xz0pq8TQ3dGqVQ1FVHuelZpAL0uNhSkk9ogYP3c40=
|
||||
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
|
||||
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
|
||||
golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug=
|
||||
golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
|
||||
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.37.0 h1:fdNQudmxPjkdUTPnLn5mdQv7Zwvbvpaxqs831goi9kQ=
|
||||
golang.org/x/sys v0.37.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
|
||||
golang.org/x/text v0.30.0 h1:yznKA/E9zq54KzlzBEAWn1NXSQ8DIp/NYMy88xJjl4k=
|
||||
golang.org/x/text v0.30.0/go.mod h1:yDdHFIX9t+tORqspjENWgzaCVXgk0yYnYuSZ8UzzBVM=
|
||||
golang.org/x/tools v0.38.0 h1:Hx2Xv8hISq8Lm16jvBZ2VQf+RLmbd7wVUsALibYI/IQ=
|
||||
golang.org/x/tools v0.38.0/go.mod h1:yEsQ/d/YK8cjh0L6rZlY8tgtlKiBNTL14pGDJPJpYQs=
|
||||
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
|
||||
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
|
||||
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
||||
golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM=
|
||||
golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY=
|
||||
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
|
||||
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
|
||||
golang.org/x/tools v0.48.0 h1:3+hClM1aLL5mjMKm5ovokw9epgRXPuu2tILgismM6RE=
|
||||
golang.org/x/tools v0.48.0/go.mod h1:08xX0orndb/F7jJxGDicx061tyd5pcMto75YMAXr6lk=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
|
||||
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
|
||||
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gorm.io/driver/sqlite v1.6.0 h1:WHRRrIiulaPiPFmDcod6prc4l2VGVWHz80KspNsxSfQ=
|
||||
gorm.io/driver/sqlite v1.6.0/go.mod h1:AO9V1qIQddBESngQUKWL9yoH93HIeA1X6V633rBwyT8=
|
||||
gorm.io/gorm v1.30.0 h1:qbT5aPv1UH8gI99OsRlvDToLxW5zR7FzS9acZDOZcgs=
|
||||
gorm.io/gorm v1.30.0/go.mod h1:8Z33v652h4//uMA76KjeDH8mJXPm1QNCYrMeatR0DOE=
|
||||
modernc.org/cc/v4 v4.24.4 h1:TFkx1s6dCkQpd6dKurBNmpo+G8Zl4Sq/ztJ+2+DEsh0=
|
||||
modernc.org/cc/v4 v4.24.4/go.mod h1:uVtb5OGqUKpoLWhqwNQo/8LwvoiEBLvZXIQ/SmO6mL0=
|
||||
modernc.org/ccgo/v4 v4.23.12 h1:UF08a38c4B+K3VoGipBrVWLFUCHd8+X20QZtFAIlQNk=
|
||||
modernc.org/ccgo/v4 v4.23.12/go.mod h1:vdN4h2WR5aEoNondUx26K7G8X+nuBscYnAEWSRmN2/0=
|
||||
modernc.org/fileutil v1.3.0 h1:gQ5SIzK3H9kdfai/5x41oQiKValumqNTDXMvKo62HvE=
|
||||
modernc.org/fileutil v1.3.0/go.mod h1:XatxS8fZi3pS8/hKG2GH/ArUogfxjpEKs3Ku3aK4JyQ=
|
||||
modernc.org/gc/v2 v2.6.1 h1:+Qf6xdG8l7B27TQ8D8lw/iFMUj1RXRBOuMUWziJOsk8=
|
||||
modernc.org/gc/v2 v2.6.1/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito=
|
||||
modernc.org/libc v1.61.8 h1:50KrjlFFoKq9ABh+bNVUf5SfVfQ4NY7CEyFBh65qc60=
|
||||
modernc.org/libc v1.61.8/go.mod h1:XloulGc0yIRM+91kbwrp7jNi/mfYPAvDOD2qwzWEij0=
|
||||
modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU=
|
||||
modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg=
|
||||
modernc.org/memory v1.8.2 h1:cL9L4bcoAObu4NkxOlKWBWtNHIsnnACGF/TbqQ6sbcI=
|
||||
modernc.org/memory v1.8.2/go.mod h1:ZbjSvMO5NQ1A2i3bWeDiVMxIorXwdClKE/0SZ+BMotU=
|
||||
modernc.org/opt v0.1.4 h1:2kNGMRiUjrp4LcaPuLY2PzUfqM/w9N23quVwhKt5Qm8=
|
||||
modernc.org/opt v0.1.4/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
|
||||
modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
|
||||
modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
|
||||
modernc.org/sqlite v1.34.4 h1:sjdARozcL5KJBvYQvLlZEmctRgW9xqIZc2ncN7PU0P8=
|
||||
modernc.org/sqlite v1.34.4/go.mod h1:3QQFCG2SEMtc2nv+Wq4cQCH7Hjcg+p/RMlS1XK+zwbk=
|
||||
modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0=
|
||||
modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A=
|
||||
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
|
||||
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
|
||||
nhooyr.io/websocket v1.8.17 h1:KEVeLJkUywCKVsnLIDlD/5gtayKp8VoCkksHCGGfT9Y=
|
||||
nhooyr.io/websocket v1.8.17/go.mod h1:rN9OFWIUwuxg4fR5tELlYC04bXYowCP9GX47ivo2l+c=
|
||||
rsc.io/qr v0.2.0 h1:6vBLea5/NRMVTz8V66gipeLycZMl/+UlFmk8DvqQ6WY=
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
// Package cli holds what every command shares: process wiring and
|
||||
// positional-argument parsing.
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"strconv"
|
||||
|
||||
"github.com/mattn/go-isatty"
|
||||
)
|
||||
|
||||
// IO is a command's terminal. Color is set when Out is a terminal, so piped
|
||||
// output stays free of ANSI codes.
|
||||
type IO struct {
|
||||
In io.Reader
|
||||
Out io.Writer
|
||||
Err io.Writer
|
||||
Color bool
|
||||
}
|
||||
|
||||
// RunFunc is a command entry point. It returns an error to exit with status 1.
|
||||
type RunFunc func(ctx context.Context, args []string, term IO) error
|
||||
|
||||
// Main runs a command against the process terminal and exits on error.
|
||||
func Main(run RunFunc) {
|
||||
term := IO{
|
||||
In: os.Stdin,
|
||||
Out: os.Stdout,
|
||||
Err: os.Stderr,
|
||||
Color: isatty.IsTerminal(os.Stdout.Fd()),
|
||||
}
|
||||
if err := run(context.Background(), os.Args[1:], term); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// WantsHelp reports whether args ask for usage. Commands parse positionals
|
||||
// by hand because the flag package would read a negative group ID such as
|
||||
// -1001234567890 as an unknown flag.
|
||||
func WantsHelp(args []string) bool {
|
||||
for _, arg := range args {
|
||||
if arg == "-h" || arg == "--help" {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// ParseGroupID parses one group ID argument.
|
||||
func ParseGroupID(value string) (int64, error) {
|
||||
id, err := strconv.ParseInt(value, 10, 64)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("invalid group id %q: must be an integer", value)
|
||||
}
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// ParseGroupIDs parses every argument as a group ID.
|
||||
func ParseGroupIDs(args []string) ([]int64, error) {
|
||||
ids := make([]int64, 0, len(args))
|
||||
for _, arg := range args {
|
||||
id, err := ParseGroupID(arg)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ids = append(ids, id)
|
||||
}
|
||||
return ids, nil
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestParseGroupIDsAcceptsNegativeIDs(t *testing.T) {
|
||||
got, err := ParseGroupIDs([]string{"-1001234567890", "-42"})
|
||||
if err != nil {
|
||||
t.Fatalf("ParseGroupIDs: %v", err)
|
||||
}
|
||||
if !reflect.DeepEqual(got, []int64{-1001234567890, -42}) {
|
||||
t.Fatalf("ParseGroupIDs = %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseGroupIDsRejectsNonInteger(t *testing.T) {
|
||||
if _, err := ParseGroupIDs([]string{"not-a-group"}); err == nil {
|
||||
t.Fatal("ParseGroupIDs accepted a non-integer")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWantsHelp(t *testing.T) {
|
||||
if !WantsHelp([]string{"-100", "--help"}) || WantsHelp([]string{"-100"}) {
|
||||
t.Fatal("WantsHelp misread arguments")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
// Package compare shows member changes for one group between two saved
|
||||
// crawls.
|
||||
package compare
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strconv"
|
||||
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/commands/cli"
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/members"
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/store"
|
||||
)
|
||||
|
||||
const usage = `usage: go run ./cmd/compare [group_id [time1 time2]]
|
||||
|
||||
Compare member changes between two crawls for one group. Without times, the
|
||||
latest two crawls are used; without a group, the first configured group.`
|
||||
|
||||
type request struct {
|
||||
groupID int64
|
||||
hasGroupID bool
|
||||
times []string
|
||||
}
|
||||
|
||||
// Run compares two crawls of one group.
|
||||
func Run(ctx context.Context, args []string, term cli.IO) error {
|
||||
if cli.WantsHelp(args) {
|
||||
fmt.Fprintln(term.Out, usage)
|
||||
return nil
|
||||
}
|
||||
req, err := parseArgs(args)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w\n%s", err, usage)
|
||||
}
|
||||
|
||||
st, err := store.OpenFromEnv(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer st.Close()
|
||||
|
||||
if !req.hasGroupID {
|
||||
cfg, err := st.LoadConfig(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.groupID = cfg.GroupIDs[0]
|
||||
}
|
||||
|
||||
before, after, err := loadPair(ctx, st, req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
added, removed := members.Diff(before.Members, after.Members)
|
||||
printReport(members.Printer{Out: term.Out, Color: term.Color}, req.groupID, before, after, added, removed)
|
||||
return nil
|
||||
}
|
||||
|
||||
func parseArgs(args []string) (request, error) {
|
||||
switch len(args) {
|
||||
case 0:
|
||||
return request{}, nil
|
||||
case 1, 3:
|
||||
groupID, err := cli.ParseGroupID(args[0])
|
||||
if err != nil {
|
||||
return request{}, err
|
||||
}
|
||||
return request{groupID: groupID, hasGroupID: true, times: args[1:]}, nil
|
||||
default:
|
||||
return request{}, errors.New("provide both time1 and time2, or omit both to compare latest two crawls")
|
||||
}
|
||||
}
|
||||
|
||||
func loadPair(ctx context.Context, st *store.Store, req request) (store.GroupExport, store.GroupExport, error) {
|
||||
if len(req.times) == 2 {
|
||||
var pair [2]store.GroupExport
|
||||
for i, runTime := range req.times {
|
||||
record, err := st.GetGroupExport(ctx, req.groupID, runTime)
|
||||
if errors.Is(err, store.ErrExportNotFound) {
|
||||
return store.GroupExport{}, store.GroupExport{}, fmt.Errorf("export not found for group %d at %s", req.groupID, runTime)
|
||||
}
|
||||
if err != nil {
|
||||
return store.GroupExport{}, store.GroupExport{}, err
|
||||
}
|
||||
pair[i] = record
|
||||
}
|
||||
return pair[0], pair[1], nil
|
||||
}
|
||||
|
||||
records, err := st.LatestGroupExports(ctx, req.groupID, 2)
|
||||
if err != nil {
|
||||
return store.GroupExport{}, store.GroupExport{}, err
|
||||
}
|
||||
if len(records) < 2 {
|
||||
return store.GroupExport{}, store.GroupExport{}, fmt.Errorf("need at least 2 exports for group %d", req.groupID)
|
||||
}
|
||||
return records[0], records[1], nil
|
||||
}
|
||||
|
||||
func printReport(out members.Printer, groupID int64, before, after store.GroupExport, added, removed []store.Member) {
|
||||
title := after.Title
|
||||
if title == "" {
|
||||
title = before.Title
|
||||
}
|
||||
label := strconv.FormatInt(groupID, 10)
|
||||
if title != "" {
|
||||
label = fmt.Sprintf("%s (%d)", title, groupID)
|
||||
}
|
||||
|
||||
out.Line(fmt.Sprintf("diff --telegram-group %q", label), "")
|
||||
out.Line("--- crawl/"+before.Time, members.Red)
|
||||
out.Line("+++ crawl/"+after.Time, members.Green)
|
||||
out.Line(fmt.Sprintf("@@ members: %d -> %d (+%d -%d) @@",
|
||||
len(before.Members), len(after.Members), len(added), len(removed)), members.Cyan)
|
||||
out.Line("", "")
|
||||
|
||||
table := members.NewTable(removed, added)
|
||||
for _, member := range removed {
|
||||
out.Line(table.Row("-", member), members.Red)
|
||||
}
|
||||
for _, member := range added {
|
||||
out.Line(table.Row("+", member), members.Green)
|
||||
}
|
||||
if len(added) == 0 && len(removed) == 0 {
|
||||
out.Line(" no membership changes.", "")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package compare
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/alicebob/miniredis/v2"
|
||||
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/commands/cli"
|
||||
)
|
||||
|
||||
func TestParseArgs(t *testing.T) {
|
||||
for _, args := range [][]string{{}, {"-100"}, {"-100", "t1", "t2"}} {
|
||||
if _, err := parseArgs(args); err != nil {
|
||||
t.Errorf("parseArgs(%v): %v", args, err)
|
||||
}
|
||||
}
|
||||
for _, args := range [][]string{{"-100", "t1"}, {"a", "b", "c", "d"}, {"x"}} {
|
||||
if _, err := parseArgs(args); err == nil {
|
||||
t.Errorf("parseArgs(%v) accepted bad input", args)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunComparesLatestTwoCrawls(t *testing.T) {
|
||||
mr := miniredis.RunT(t)
|
||||
t.Setenv("REDIS_URL", "redis://"+mr.Addr())
|
||||
mr.Set("telegram-export:group:-100:20260701000000",
|
||||
`{"group_id":-100,"title":"Team","time":"20260701000000","members":[{"id":1,"username":"amy"},{"id":2,"username":null}]}`)
|
||||
mr.Set("telegram-export:group:-100:20260702000000",
|
||||
`{"group_id":-100,"title":"Team","time":"20260702000000","members":[{"id":2},{"id":3,"username":"cat","first_name":"Cat"}]}`)
|
||||
|
||||
var out bytes.Buffer
|
||||
if err := Run(context.Background(), []string{"-100"}, cli.IO{Out: &out}); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
|
||||
want := strings.Join([]string{
|
||||
`diff --telegram-group "Team (-100)"`,
|
||||
`--- crawl/20260701000000`,
|
||||
`+++ crawl/20260702000000`,
|
||||
`@@ members: 2 -> 2 (+1 -1) @@`,
|
||||
``,
|
||||
`- 1 @amy`,
|
||||
`+ 3 @cat Cat`,
|
||||
``,
|
||||
}, "\n")
|
||||
if out.String() != want {
|
||||
t.Fatalf("output:\n%s\nwant:\n%s", out.String(), want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunNeedsTwoCrawls(t *testing.T) {
|
||||
mr := miniredis.RunT(t)
|
||||
t.Setenv("REDIS_URL", "redis://"+mr.Addr())
|
||||
|
||||
err := Run(context.Background(), []string{"-100"}, cli.IO{Out: &bytes.Buffer{}})
|
||||
if err == nil || !strings.Contains(err.Error(), "need at least 2 exports for group -100") {
|
||||
t.Fatalf("Run error = %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
// Package comparegroups fetches two groups live from Telegram and compares
|
||||
// their members. It saves no crawl to Redis.
|
||||
package comparegroups
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"github.com/gotd/td/tg"
|
||||
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/commands/cli"
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/members"
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/prompt"
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/store"
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/telegram"
|
||||
)
|
||||
|
||||
const usage = `usage: go run ./cmd/compare-groups <group_a> <group_b>
|
||||
|
||||
Fetch two Telegram groups live and compare their members. No crawl is saved
|
||||
to Redis; Redis only supplies the Telegram config and login session.`
|
||||
|
||||
// Run fetches both groups and prints who is in both, only A, and only B.
|
||||
func Run(ctx context.Context, args []string, term cli.IO) error {
|
||||
if cli.WantsHelp(args) {
|
||||
fmt.Fprintln(term.Out, usage)
|
||||
return nil
|
||||
}
|
||||
if len(args) != 2 {
|
||||
return errors.New(usage)
|
||||
}
|
||||
groupIDs, err := cli.ParseGroupIDs(args)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w\n%s", err, usage)
|
||||
}
|
||||
|
||||
st, err := store.OpenFromEnv(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer st.Close()
|
||||
|
||||
cfg, err := st.LoadConfig(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
ask := prompt.New(term.In, term.Out)
|
||||
return telegram.Run(ctx, cfg, st.SessionStorage(), ask, func(ctx context.Context, api *tg.Client) error {
|
||||
fetcher := telegram.NewFetcher(api)
|
||||
var groups [2]telegram.Group
|
||||
for i, groupID := range groupIDs {
|
||||
group, err := fetcher.FetchGroup(ctx, groupID, func(title string) {
|
||||
fmt.Fprintf(term.Out, "fetching %s (ID: %d)\n", title, groupID)
|
||||
})
|
||||
if errors.Is(err, telegram.ErrNotGroup) {
|
||||
return fmt.Errorf("%d is not a group", groupID)
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("error accessing group %d: %w", groupID, err)
|
||||
}
|
||||
groups[i] = group
|
||||
}
|
||||
|
||||
fmt.Fprintln(term.Out)
|
||||
printReport(members.Printer{Out: term.Out, Color: term.Color}, groups[0], groups[1])
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
func printReport(out members.Printer, a, b telegram.Group) {
|
||||
onlyA, onlyB, both := members.Split(a.Members, b.Members)
|
||||
|
||||
out.Line(fmt.Sprintf("diff --telegram-groups %q %q", label(a), label(b)), "")
|
||||
out.Line("--- "+label(a), members.Red)
|
||||
out.Line("+++ "+label(b), members.Green)
|
||||
out.Line(fmt.Sprintf("@@ members: %d vs %d (both %d, -%d only first, +%d only second) @@",
|
||||
len(a.Members), len(b.Members), len(both), len(onlyA), len(onlyB)), members.Cyan)
|
||||
out.Line("", "")
|
||||
|
||||
table := members.NewTable(onlyA, onlyB, both)
|
||||
for _, member := range both {
|
||||
out.Line(table.Row(" ", member), "")
|
||||
}
|
||||
for _, member := range onlyA {
|
||||
out.Line(table.Row("-", member), members.Red)
|
||||
}
|
||||
for _, member := range onlyB {
|
||||
out.Line(table.Row("+", member), members.Green)
|
||||
}
|
||||
if len(a.Members) == 0 && len(b.Members) == 0 {
|
||||
out.Line(" both groups have no visible members.", "")
|
||||
}
|
||||
}
|
||||
|
||||
func label(group telegram.Group) string {
|
||||
return fmt.Sprintf("%s (%d)", group.Title, group.ID)
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
package comparegroups
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/commands/cli"
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/members"
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/store"
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/telegram"
|
||||
)
|
||||
|
||||
func TestPrintReport(t *testing.T) {
|
||||
a := telegram.Group{ID: -101, Title: "A", Members: []store.Member{
|
||||
{ID: 1, Username: "amy"}, {ID: 2, Username: "bob"},
|
||||
}}
|
||||
b := telegram.Group{ID: -202, Title: "B", Members: []store.Member{
|
||||
{ID: 2, Username: "bob"}, {ID: 3, Username: "cat"},
|
||||
}}
|
||||
|
||||
var out bytes.Buffer
|
||||
printReport(members.Printer{Out: &out}, a, b)
|
||||
|
||||
want := strings.Join([]string{
|
||||
`diff --telegram-groups "A (-101)" "B (-202)"`,
|
||||
`--- A (-101)`,
|
||||
`+++ B (-202)`,
|
||||
`@@ members: 2 vs 2 (both 1, -1 only first, +1 only second) @@`,
|
||||
``,
|
||||
` 2 @bob`,
|
||||
`- 1 @amy`,
|
||||
`+ 3 @cat`,
|
||||
``,
|
||||
}, "\n")
|
||||
if out.String() != want {
|
||||
t.Fatalf("output:\n%s\nwant:\n%s", out.String(), want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunRequiresTwoGroupIDsBeforeRedis(t *testing.T) {
|
||||
t.Setenv("REDIS_URL", "")
|
||||
for _, args := range [][]string{{"-100"}, {"-100", "x"}} {
|
||||
err := Run(context.Background(), args, cli.IO{Out: &bytes.Buffer{}})
|
||||
if err == nil || strings.Contains(err.Error(), "REDIS_URL") {
|
||||
t.Errorf("Run(%v) error = %v, want a usage error", args, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
// Package configure stores the Telegram crawl config in Redis.
|
||||
package configure
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/commands/cli"
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/prompt"
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/store"
|
||||
)
|
||||
|
||||
// Run asks for any value not given as a flag, then saves the config.
|
||||
func Run(ctx context.Context, args []string, term cli.IO) error {
|
||||
flags := flag.NewFlagSet("configure", flag.ContinueOnError)
|
||||
flags.SetOutput(term.Err)
|
||||
flags.Usage = func() {
|
||||
fmt.Fprintln(term.Err, "usage: go run ./cmd/configure [--api-id ID] [--api-hash HASH] [--phone PHONE] [--groups ID,ID]")
|
||||
fmt.Fprintln(term.Err, "\nStore Telegram crawl config in Redis.")
|
||||
flags.PrintDefaults()
|
||||
}
|
||||
apiID := flags.Int("api-id", 0, "Telegram API ID")
|
||||
apiHash := flags.String("api-hash", "", "Telegram API hash")
|
||||
phone := flags.String("phone", "", "Telegram account phone")
|
||||
groups := flags.String("groups", "", "Comma-separated Telegram group IDs to crawl")
|
||||
if err := flags.Parse(args); err != nil {
|
||||
if errors.Is(err, flag.ErrHelp) {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
ask := prompt.New(term.In, term.Out)
|
||||
cfg, err := collect(ask, *apiID, *apiHash, *phone, *groups)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
st, err := store.OpenFromEnv(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer st.Close()
|
||||
|
||||
if err := st.SaveConfig(ctx, cfg); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(term.Out, "saved Redis config for %s: %d group(s).\n", cfg.Phone, len(cfg.GroupIDs))
|
||||
return nil
|
||||
}
|
||||
|
||||
func collect(ask *prompt.Prompter, apiID int, apiHash, phone, groups string) (store.AppConfig, error) {
|
||||
var err error
|
||||
if apiID == 0 {
|
||||
raw, err := required(ask.Line, "API ID")
|
||||
if err != nil {
|
||||
return store.AppConfig{}, err
|
||||
}
|
||||
if apiID, err = strconv.Atoi(raw); err != nil {
|
||||
return store.AppConfig{}, errors.New("api id must be an integer")
|
||||
}
|
||||
}
|
||||
if apiHash == "" {
|
||||
if apiHash, err = required(ask.Secret, "API hash"); err != nil {
|
||||
return store.AppConfig{}, err
|
||||
}
|
||||
}
|
||||
if phone == "" {
|
||||
if phone, err = required(ask.Line, "Phone"); err != nil {
|
||||
return store.AppConfig{}, err
|
||||
}
|
||||
}
|
||||
if groups == "" {
|
||||
if groups, err = required(ask.Line, "Group IDs"); err != nil {
|
||||
return store.AppConfig{}, err
|
||||
}
|
||||
}
|
||||
groupIDs, err := ParseGroupList(groups)
|
||||
if err != nil {
|
||||
return store.AppConfig{}, err
|
||||
}
|
||||
return store.AppConfig{APIID: apiID, APIHash: apiHash, Phone: phone, GroupIDs: groupIDs}, nil
|
||||
}
|
||||
|
||||
func required(read func(string) (string, error), label string) (string, error) {
|
||||
value, err := read(label)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if value == "" {
|
||||
return "", fmt.Errorf("%s is required", strings.ToLower(label))
|
||||
}
|
||||
return value, nil
|
||||
}
|
||||
|
||||
// ParseGroupList parses comma-separated group IDs, ignoring empty entries.
|
||||
func ParseGroupList(value string) ([]int64, error) {
|
||||
var ids []int64
|
||||
for _, part := range strings.Split(value, ",") {
|
||||
part = strings.TrimSpace(part)
|
||||
if part == "" {
|
||||
continue
|
||||
}
|
||||
id, err := strconv.ParseInt(part, 10, 64)
|
||||
if err != nil {
|
||||
return nil, errors.New("group ids must be comma-separated integers")
|
||||
}
|
||||
ids = append(ids, id)
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
return nil, errors.New("at least one group id is required")
|
||||
}
|
||||
return ids, nil
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
package configure
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/prompt"
|
||||
)
|
||||
|
||||
func TestParseGroupList(t *testing.T) {
|
||||
got, err := ParseGroupList(" -1001, ,-2 ")
|
||||
if err != nil || !reflect.DeepEqual(got, []int64{-1001, -2}) {
|
||||
t.Fatalf("ParseGroupList = %v, %v", got, err)
|
||||
}
|
||||
for _, bad := range []string{"", " , ", "abc"} {
|
||||
if _, err := ParseGroupList(bad); err == nil {
|
||||
t.Errorf("ParseGroupList(%q) accepted bad input", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCollectPromptsOnlyForMissingValues(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
ask := prompt.New(strings.NewReader("+1555\n-100,-200\n"), &out)
|
||||
|
||||
cfg, err := collect(ask, 123, "hash", "", "")
|
||||
if err != nil {
|
||||
t.Fatalf("collect: %v", err)
|
||||
}
|
||||
if cfg.Phone != "+1555" || !reflect.DeepEqual(cfg.GroupIDs, []int64{-100, -200}) || cfg.APIID != 123 {
|
||||
t.Fatalf("collect = %+v", cfg)
|
||||
}
|
||||
if strings.Contains(out.String(), "API ID") || !strings.Contains(out.String(), "Phone: ") {
|
||||
t.Fatalf("unexpected prompts: %q", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCollectRejectsBadAPIID(t *testing.T) {
|
||||
ask := prompt.New(strings.NewReader("abc\n"), &bytes.Buffer{})
|
||||
if _, err := collect(ask, 0, "", "", ""); err == nil || !strings.Contains(err.Error(), "integer") {
|
||||
t.Fatalf("collect error = %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
// Package crawl fetches group members from Telegram and saves each group as
|
||||
// one export in Redis.
|
||||
package crawl
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"github.com/gotd/td/tg"
|
||||
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/commands/cli"
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/prompt"
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/store"
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/telegram"
|
||||
)
|
||||
|
||||
const usage = `usage: go run ./cmd/crawl [group_id ...]
|
||||
|
||||
Export members from Telegram groups. Group IDs given here replace the
|
||||
configured list for this run only.`
|
||||
|
||||
// Run crawls the given groups, or the configured ones when none are given.
|
||||
func Run(ctx context.Context, args []string, term cli.IO) error {
|
||||
if cli.WantsHelp(args) {
|
||||
fmt.Fprintln(term.Out, usage)
|
||||
return nil
|
||||
}
|
||||
// Parse before touching Redis or Telegram, so bad input fails offline.
|
||||
groupIDs, err := cli.ParseGroupIDs(args)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w\n%s", err, usage)
|
||||
}
|
||||
|
||||
st, err := store.OpenFromEnv(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer st.Close()
|
||||
|
||||
cfg, err := st.LoadConfig(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(groupIDs) == 0 {
|
||||
groupIDs = cfg.GroupIDs
|
||||
}
|
||||
|
||||
ask := prompt.New(term.In, term.Out)
|
||||
return telegram.Run(ctx, cfg, st.SessionStorage(), ask, func(ctx context.Context, api *tg.Client) error {
|
||||
runTime := store.NewRunTime()
|
||||
saved := crawlGroups(ctx, term, telegram.NewFetcher(api), st, groupIDs, runTime)
|
||||
fmt.Fprintf(term.Out, "saved run %s: %d group(s) to Redis.\n", runTime, saved)
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// crawlGroups saves every group it can read. One failing group is reported
|
||||
// and skipped so the rest of the run still lands.
|
||||
func crawlGroups(
|
||||
ctx context.Context,
|
||||
term cli.IO,
|
||||
fetcher *telegram.Fetcher,
|
||||
st *store.Store,
|
||||
groupIDs []int64,
|
||||
runTime string,
|
||||
) int {
|
||||
saved := 0
|
||||
for _, groupID := range groupIDs {
|
||||
group, err := fetcher.FetchGroup(ctx, groupID, func(title string) {
|
||||
fmt.Fprintf(term.Out, "exporting %s (ID: %d)\n", title, groupID)
|
||||
})
|
||||
if errors.Is(err, telegram.ErrNotGroup) {
|
||||
fmt.Fprintf(term.Out, "skip %d because it is not a group.\n", groupID)
|
||||
continue
|
||||
}
|
||||
if err == nil {
|
||||
err = st.SaveGroupExport(ctx, store.GroupExport{
|
||||
GroupID: groupID,
|
||||
Title: group.Title,
|
||||
Time: runTime,
|
||||
Members: group.Members,
|
||||
})
|
||||
}
|
||||
if err != nil {
|
||||
fmt.Fprintf(term.Out, "error accessing group %d: %v\n", groupID, err)
|
||||
continue
|
||||
}
|
||||
saved++
|
||||
fmt.Fprintf(term.Out, "export %s done.\n", group.Title)
|
||||
}
|
||||
return saved
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
package crawl
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/commands/cli"
|
||||
)
|
||||
|
||||
func TestRunRejectsBadGroupIDBeforeRedis(t *testing.T) {
|
||||
t.Setenv("REDIS_URL", "")
|
||||
var out bytes.Buffer
|
||||
|
||||
err := Run(context.Background(), []string{"not-a-group"}, cli.IO{Out: &out, Err: &out})
|
||||
if err == nil || !strings.Contains(err.Error(), `invalid group id "not-a-group"`) {
|
||||
t.Fatalf("Run error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunHelp(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
if err := Run(context.Background(), []string{"-h"}, cli.IO{Out: &out}); err != nil {
|
||||
t.Fatalf("Run -h: %v", err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "go run ./cmd/crawl") {
|
||||
t.Fatalf("help = %q", out.String())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
// Package export writes one saved group crawl from Redis to a CSV file.
|
||||
package export
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/commands/cli"
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/store"
|
||||
)
|
||||
|
||||
const usage = `usage: go run ./cmd/export [group_id [timecrawl]]
|
||||
|
||||
Export a Redis-backed Telegram group crawl to CSV. Without timecrawl
|
||||
(yyyymmddhhmmss), the latest crawl is used; without a group, the first
|
||||
configured group.`
|
||||
|
||||
// outputDir is where CSV files land; a var so tests can redirect it.
|
||||
var outputDir = "output"
|
||||
|
||||
// Run exports one crawl to output/<group-id>-<yyyymmddhhmmss>.csv.
|
||||
func Run(ctx context.Context, args []string, term cli.IO) error {
|
||||
if cli.WantsHelp(args) {
|
||||
fmt.Fprintln(term.Out, usage)
|
||||
return nil
|
||||
}
|
||||
if len(args) > 2 {
|
||||
return errors.New(usage)
|
||||
}
|
||||
var (
|
||||
groupID int64
|
||||
hasGroupID bool
|
||||
runTime string
|
||||
err error
|
||||
)
|
||||
if len(args) >= 1 {
|
||||
if groupID, err = cli.ParseGroupID(args[0]); err != nil {
|
||||
return fmt.Errorf("%w\n%s", err, usage)
|
||||
}
|
||||
hasGroupID = true
|
||||
}
|
||||
if len(args) == 2 {
|
||||
runTime = args[1]
|
||||
if err := validateRunTime(runTime); err != nil {
|
||||
return fmt.Errorf("%w\n%s", err, usage)
|
||||
}
|
||||
}
|
||||
|
||||
st, err := store.OpenFromEnv(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer st.Close()
|
||||
|
||||
record, groupID, runTime, err := loadRecord(ctx, st, groupID, hasGroupID, runTime)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
rows, err := validateRecord(record, groupID, runTime)
|
||||
if err == nil {
|
||||
var path string
|
||||
if path, err = writeCSV(outputDir, groupID, runTime, rows); err == nil {
|
||||
fmt.Fprintf(term.Out, "exported %d members to %s\n", len(rows), path)
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("could not export group %d at %s: %w", groupID, runTime, err)
|
||||
}
|
||||
|
||||
// loadRecord resolves the default group and latest time, then reads that
|
||||
// export. It returns the resolved group ID and time with the record.
|
||||
func loadRecord(ctx context.Context, st *store.Store, groupID int64, hasGroupID bool, runTime string) (store.GroupExport, int64, string, error) {
|
||||
if !hasGroupID {
|
||||
cfg, err := st.LoadConfig(ctx)
|
||||
if err != nil {
|
||||
return store.GroupExport{}, 0, "", fmt.Errorf("could not read export configuration or history: %w", err)
|
||||
}
|
||||
groupID = cfg.GroupIDs[0]
|
||||
}
|
||||
if runTime == "" {
|
||||
times, err := st.GroupExportTimes(ctx, groupID)
|
||||
if err != nil {
|
||||
return store.GroupExport{}, 0, "", fmt.Errorf("could not read export configuration or history: %w", err)
|
||||
}
|
||||
if len(times) == 0 {
|
||||
return store.GroupExport{}, 0, "", fmt.Errorf("no exports found for group %d", groupID)
|
||||
}
|
||||
runTime = times[len(times)-1]
|
||||
}
|
||||
|
||||
record, err := st.GetGroupExport(ctx, groupID, runTime)
|
||||
if errors.Is(err, store.ErrExportNotFound) {
|
||||
return store.GroupExport{}, 0, "", fmt.Errorf("export not found or invalid for group %d at %s", groupID, runTime)
|
||||
}
|
||||
if err != nil {
|
||||
return store.GroupExport{}, 0, "", fmt.Errorf("could not read export configuration or history: %w", err)
|
||||
}
|
||||
return record, groupID, runTime, nil
|
||||
}
|
||||
|
||||
func validateRunTime(value string) error {
|
||||
if !store.IsRunTime(value) {
|
||||
return errors.New("timecrawl must use yyyymmddhhmmss format")
|
||||
}
|
||||
if _, err := time.Parse(store.RunTimeLayout, value); err != nil {
|
||||
return errors.New("timecrawl must be a valid date and time")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
package export
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/csv"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/alicebob/miniredis/v2"
|
||||
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/commands/cli"
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/store"
|
||||
)
|
||||
|
||||
func setup(t *testing.T) (*miniredis.Miniredis, string) {
|
||||
t.Helper()
|
||||
mr := miniredis.RunT(t)
|
||||
t.Setenv("REDIS_URL", "redis://"+mr.Addr())
|
||||
dir := filepath.Join(t.TempDir(), "output")
|
||||
old := outputDir
|
||||
outputDir = dir
|
||||
t.Cleanup(func() { outputDir = old })
|
||||
return mr, dir
|
||||
}
|
||||
|
||||
func TestRunExportsLatestCrawl(t *testing.T) {
|
||||
mr, dir := setup(t)
|
||||
mr.Set("telegram-export:group:-100:20260701000000", `{"group_id":-100,"time":"20260701000000","members":[]}`)
|
||||
mr.Set("telegram-export:group:-100:20260702000000",
|
||||
`{"group_id":-100,"time":"20260702000000","members":[{"id":1,"username":"amy","first_name":"Amy","last_name":"Lee"},{"id":2,"username":null,"first_name":"=SUM(A1)","last_name":null}]}`)
|
||||
|
||||
var out bytes.Buffer
|
||||
if err := Run(context.Background(), []string{"-100"}, cli.IO{Out: &out}); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
|
||||
path := filepath.Join(dir, "-100-20260702000000.csv")
|
||||
if !strings.Contains(out.String(), "exported 2 members to "+path) {
|
||||
t.Fatalf("output = %q", out.String())
|
||||
}
|
||||
file, err := os.Open(path)
|
||||
if err != nil {
|
||||
t.Fatalf("open CSV: %v", err)
|
||||
}
|
||||
defer file.Close()
|
||||
rows, err := csv.NewReader(file).ReadAll()
|
||||
if err != nil {
|
||||
t.Fatalf("read CSV: %v", err)
|
||||
}
|
||||
want := [][]string{
|
||||
{"id", "username", "first_name", "last_name"},
|
||||
{"1", "amy", "Amy", "Lee"},
|
||||
{"2", "", "'=SUM(A1)", ""},
|
||||
}
|
||||
if !reflect.DeepEqual(rows, want) {
|
||||
t.Fatalf("rows = %v, want %v", rows, want)
|
||||
}
|
||||
|
||||
for p, mode := range map[string]os.FileMode{dir: 0o700, path: 0o600} {
|
||||
if info, err := os.Stat(p); err != nil || info.Mode().Perm() != mode {
|
||||
t.Errorf("%s mode = %v, %v; want %v", p, info.Mode().Perm(), err, mode)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunRejectsMismatchedRecord(t *testing.T) {
|
||||
mr, _ := setup(t)
|
||||
mr.Set("telegram-export:group:-100:20260702000000", `{"group_id":-999,"time":"20260702000000","members":[]}`)
|
||||
|
||||
err := Run(context.Background(), []string{"-100", "20260702000000"}, cli.IO{Out: &bytes.Buffer{}})
|
||||
if err == nil || !strings.Contains(err.Error(), "group ID does not match its key") {
|
||||
t.Fatalf("Run error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunReportsMissingExports(t *testing.T) {
|
||||
setup(t)
|
||||
err := Run(context.Background(), []string{"-100"}, cli.IO{Out: &bytes.Buffer{}})
|
||||
if err == nil || err.Error() != "no exports found for group -100" {
|
||||
t.Fatalf("Run error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRunTime(t *testing.T) {
|
||||
tests := map[string]string{
|
||||
"20260724120000": "",
|
||||
"2026072412": "yyyymmddhhmmss format",
|
||||
"20260230120000": "valid date and time",
|
||||
"20261324000000": "valid date and time",
|
||||
}
|
||||
for value, want := range tests {
|
||||
err := validateRunTime(value)
|
||||
if (want == "") != (err == nil) || (err != nil && !strings.Contains(err.Error(), want)) {
|
||||
t.Errorf("validateRunTime(%q) = %v, want %q", value, err, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestEscapeFormula(t *testing.T) {
|
||||
tests := map[string]string{
|
||||
"": "",
|
||||
"alice": "alice",
|
||||
"=1+1": "'=1+1",
|
||||
" @cmd": "' @cmd",
|
||||
"-5": "'-5",
|
||||
"\tx": "'\tx",
|
||||
"\nx": "'\nx",
|
||||
}
|
||||
for in, want := range tests {
|
||||
if got := escapeFormula(in); got != want {
|
||||
t.Errorf("escapeFormula(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRecordRejectsMissingMemberID(t *testing.T) {
|
||||
record := store.GroupExport{GroupID: -1, Time: "t", Members: []store.Member{{Username: "x"}}}
|
||||
if _, err := validateRecord(record, -1, "t"); err == nil || !strings.Contains(err.Error(), "invalid ID") {
|
||||
t.Fatalf("validateRecord error = %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
package export
|
||||
|
||||
import (
|
||||
"encoding/csv"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode"
|
||||
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/store"
|
||||
)
|
||||
|
||||
var csvHeader = []string{"id", "username", "first_name", "last_name"}
|
||||
|
||||
// validateRecord turns members into CSV rows, rejecting a crawl whose key
|
||||
// and body disagree or whose members lack an ID.
|
||||
func validateRecord(record store.GroupExport, groupID int64, runTime string) ([][]string, error) {
|
||||
if record.GroupID != groupID {
|
||||
return nil, errors.New("Redis export group ID does not match its key")
|
||||
}
|
||||
if record.Time != runTime {
|
||||
return nil, errors.New("Redis export time does not match its key")
|
||||
}
|
||||
rows := make([][]string, 0, len(record.Members))
|
||||
for i, member := range record.Members {
|
||||
if member.ID == 0 {
|
||||
return nil, fmt.Errorf("member %d has an invalid ID", i)
|
||||
}
|
||||
rows = append(rows, []string{
|
||||
strconv.FormatInt(member.ID, 10),
|
||||
escapeFormula(member.Username),
|
||||
escapeFormula(member.FirstName),
|
||||
escapeFormula(member.LastName),
|
||||
})
|
||||
}
|
||||
return rows, nil
|
||||
}
|
||||
|
||||
// escapeFormula prefixes an apostrophe to Telegram text a spreadsheet would
|
||||
// run as a formula.
|
||||
func escapeFormula(value string) string {
|
||||
if value == "" {
|
||||
return value
|
||||
}
|
||||
if strings.ContainsRune("\t\r\n", rune(value[0])) {
|
||||
return "'" + value
|
||||
}
|
||||
trimmed := strings.TrimLeftFunc(value, unicode.IsSpace)
|
||||
if trimmed != "" && strings.ContainsRune("=+-@\t\r", rune(trimmed[0])) {
|
||||
return "'" + value
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
// writeCSV writes through a temp file and renames it, so a failed export
|
||||
// never leaves a half-written CSV. Member data is personal, so the directory
|
||||
// is 0700 and the file 0600.
|
||||
func writeCSV(dir string, groupID int64, runTime string, rows [][]string) (path string, err error) {
|
||||
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := os.Chmod(dir, 0o700); err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
path = filepath.Join(dir, fmt.Sprintf("%d-%s.csv", groupID, runTime))
|
||||
tmp, err := os.CreateTemp(dir, "."+filepath.Base(path)+".")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer func() {
|
||||
if err != nil {
|
||||
_ = tmp.Close()
|
||||
_ = os.Remove(tmp.Name())
|
||||
}
|
||||
}()
|
||||
|
||||
if err = tmp.Chmod(0o600); err != nil {
|
||||
return "", err
|
||||
}
|
||||
writer := csv.NewWriter(tmp)
|
||||
writer.UseCRLF = true
|
||||
if err = writer.Write(csvHeader); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = writer.WriteAll(rows); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = tmp.Close(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = os.Rename(tmp.Name(), path); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return path, nil
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
package members
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/store"
|
||||
)
|
||||
|
||||
// ANSI colors for git-diff-style output.
|
||||
const (
|
||||
Red = "\033[31m"
|
||||
Green = "\033[32m"
|
||||
Cyan = "\033[36m"
|
||||
reset = "\033[0m"
|
||||
)
|
||||
|
||||
// Printer writes diff lines, coloring them only when Color is set (stdout is
|
||||
// a terminal), so piped output stays plain.
|
||||
type Printer struct {
|
||||
Out io.Writer
|
||||
Color bool
|
||||
}
|
||||
|
||||
// Line prints text in the given color, or plain when color is empty.
|
||||
func (p Printer) Line(text, color string) {
|
||||
if p.Color && color != "" {
|
||||
text = color + text + reset
|
||||
}
|
||||
fmt.Fprintln(p.Out, text)
|
||||
}
|
||||
|
||||
// Table aligns member rows: ID, @username, and full name.
|
||||
type Table struct {
|
||||
widths [3]int
|
||||
}
|
||||
|
||||
// NewTable sizes the columns to fit every given member.
|
||||
func NewTable(lists ...[]store.Member) Table {
|
||||
t := Table{widths: [3]int{2, 8, 4}}
|
||||
for _, list := range lists {
|
||||
for _, member := range list {
|
||||
for i, value := range columns(member) {
|
||||
t.widths[i] = max(t.widths[i], utf8.RuneCountInString(value))
|
||||
}
|
||||
}
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
// Row formats one member behind a one-character prefix.
|
||||
func (t Table) Row(prefix string, member store.Member) string {
|
||||
cols := columns(member)
|
||||
row := prefix + " " +
|
||||
pad(cols[0], t.widths[0]) + " " +
|
||||
pad(cols[1], t.widths[1]) + " " +
|
||||
pad(cols[2], t.widths[2])
|
||||
return strings.TrimRight(row, " ")
|
||||
}
|
||||
|
||||
func columns(member store.Member) [3]string {
|
||||
username := ""
|
||||
if member.Username != "" {
|
||||
username = "@" + member.Username
|
||||
}
|
||||
var names []string
|
||||
for _, part := range []string{member.FirstName, member.LastName} {
|
||||
if part != "" {
|
||||
names = append(names, part)
|
||||
}
|
||||
}
|
||||
return [3]string{
|
||||
strconv.FormatInt(member.ID, 10),
|
||||
username,
|
||||
strings.Join(names, " "),
|
||||
}
|
||||
}
|
||||
|
||||
func pad(value string, width int) string {
|
||||
return value + strings.Repeat(" ", max(0, width-utf8.RuneCountInString(value)))
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
// Package members compares member lists and formats them as diff rows. It
|
||||
// stays pure: no Redis and no network.
|
||||
package members
|
||||
|
||||
import (
|
||||
"sort"
|
||||
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/store"
|
||||
)
|
||||
|
||||
// Diff returns members added in after and removed from before, matched by
|
||||
// Telegram member ID and sorted by ID.
|
||||
func Diff(before, after []store.Member) (added, removed []store.Member) {
|
||||
onlyBefore, onlyAfter, _ := Split(before, after)
|
||||
return onlyAfter, onlyBefore
|
||||
}
|
||||
|
||||
// Split returns members only in a, only in b, and in both, matched by
|
||||
// Telegram member ID and sorted by ID. Members in both come from b.
|
||||
func Split(a, b []store.Member) (onlyA, onlyB, both []store.Member) {
|
||||
aByID := index(a)
|
||||
bByID := index(b)
|
||||
|
||||
for id, member := range aByID {
|
||||
if _, ok := bByID[id]; !ok {
|
||||
onlyA = append(onlyA, member)
|
||||
}
|
||||
}
|
||||
for id, member := range bByID {
|
||||
if _, ok := aByID[id]; ok {
|
||||
both = append(both, member)
|
||||
} else {
|
||||
onlyB = append(onlyB, member)
|
||||
}
|
||||
}
|
||||
sortByID(onlyA)
|
||||
sortByID(onlyB)
|
||||
sortByID(both)
|
||||
return onlyA, onlyB, both
|
||||
}
|
||||
|
||||
// index keys members by ID; ID 0 is never a real Telegram user.
|
||||
func index(list []store.Member) map[int64]store.Member {
|
||||
byID := make(map[int64]store.Member, len(list))
|
||||
for _, member := range list {
|
||||
if member.ID != 0 {
|
||||
byID[member.ID] = member
|
||||
}
|
||||
}
|
||||
return byID
|
||||
}
|
||||
|
||||
func sortByID(list []store.Member) {
|
||||
sort.Slice(list, func(i, j int) bool { return list[i].ID < list[j].ID })
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
package members
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"reflect"
|
||||
"testing"
|
||||
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/store"
|
||||
)
|
||||
|
||||
func ids(list []store.Member) []int64 {
|
||||
out := []int64{}
|
||||
for _, member := range list {
|
||||
out = append(out, member.ID)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func membersWithIDs(values ...int64) []store.Member {
|
||||
var list []store.Member
|
||||
for _, id := range values {
|
||||
list = append(list, store.Member{ID: id})
|
||||
}
|
||||
return list
|
||||
}
|
||||
|
||||
func TestSplit(t *testing.T) {
|
||||
onlyA, onlyB, both := Split(membersWithIDs(3, 1, 2), membersWithIDs(4, 2, 3))
|
||||
|
||||
if !reflect.DeepEqual(ids(onlyA), []int64{1}) {
|
||||
t.Errorf("onlyA = %v", ids(onlyA))
|
||||
}
|
||||
if !reflect.DeepEqual(ids(onlyB), []int64{4}) {
|
||||
t.Errorf("onlyB = %v", ids(onlyB))
|
||||
}
|
||||
if !reflect.DeepEqual(ids(both), []int64{2, 3}) {
|
||||
t.Errorf("both = %v", ids(both))
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiff(t *testing.T) {
|
||||
added, removed := Diff(membersWithIDs(1, 2, 0), membersWithIDs(2, 3))
|
||||
|
||||
if !reflect.DeepEqual(ids(added), []int64{3}) || !reflect.DeepEqual(ids(removed), []int64{1}) {
|
||||
t.Fatalf("Diff added=%v removed=%v", ids(added), ids(removed))
|
||||
}
|
||||
}
|
||||
|
||||
func TestTableRowAlignsAndTrims(t *testing.T) {
|
||||
amy := store.Member{ID: 1, Username: "amy", FirstName: "Amy", LastName: "Lê"}
|
||||
bob := store.Member{ID: 22, FirstName: "Bob"}
|
||||
table := NewTable([]store.Member{amy, bob})
|
||||
|
||||
if got, want := table.Row("-", amy), "- 1 @amy Amy Lê"; got != want {
|
||||
t.Errorf("Row(amy) = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := table.Row("+", bob), "+ 22 Bob"; got != want {
|
||||
t.Errorf("Row(bob) = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrinterColorsOnlyWhenEnabled(t *testing.T) {
|
||||
var plain, colored bytes.Buffer
|
||||
Printer{Out: &plain}.Line("x", Red)
|
||||
Printer{Out: &colored, Color: true}.Line("x", Red)
|
||||
|
||||
if plain.String() != "x\n" {
|
||||
t.Errorf("plain = %q", plain.String())
|
||||
}
|
||||
if colored.String() != Red+"x"+reset+"\n" {
|
||||
t.Errorf("colored = %q", colored.String())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
// Package prompt reads answers from the terminal.
|
||||
package prompt
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"golang.org/x/term"
|
||||
)
|
||||
|
||||
// Prompter asks questions on out and reads answers from in.
|
||||
type Prompter struct {
|
||||
in *bufio.Reader
|
||||
fd int
|
||||
tty bool
|
||||
out io.Writer
|
||||
}
|
||||
|
||||
// New returns a Prompter. Secrets are read without echo when in is a
|
||||
// terminal.
|
||||
func New(in io.Reader, out io.Writer) *Prompter {
|
||||
p := &Prompter{in: bufio.NewReader(in), out: out}
|
||||
if f, ok := in.(*os.File); ok && term.IsTerminal(int(f.Fd())) {
|
||||
p.fd, p.tty = int(f.Fd()), true
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// Line asks for one line of text and trims it.
|
||||
func (p *Prompter) Line(label string) (string, error) {
|
||||
fmt.Fprintf(p.out, "%s: ", label)
|
||||
line, err := p.in.ReadString('\n')
|
||||
if err != nil && !(errors.Is(err, io.EOF) && line != "") {
|
||||
return "", fmt.Errorf("read %s: %w", strings.ToLower(label), err)
|
||||
}
|
||||
return strings.TrimSpace(line), nil
|
||||
}
|
||||
|
||||
// Secret asks for a value without echoing it on a terminal.
|
||||
func (p *Prompter) Secret(label string) (string, error) {
|
||||
if !p.tty {
|
||||
return p.Line(label)
|
||||
}
|
||||
fmt.Fprintf(p.out, "%s: ", label)
|
||||
raw, err := term.ReadPassword(p.fd)
|
||||
fmt.Fprintln(p.out)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("read %s: %w", strings.ToLower(label), err)
|
||||
}
|
||||
return strings.TrimSpace(string(raw)), nil
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
// AppConfig is the Telegram crawl config stored as one JSON key.
|
||||
type AppConfig struct {
|
||||
APIID int `json:"api_id"`
|
||||
APIHash string `json:"api_hash"`
|
||||
Phone string `json:"phone"`
|
||||
GroupIDs []int64 `json:"group_ids"`
|
||||
}
|
||||
|
||||
func configKey() string { return key("config") }
|
||||
|
||||
// LoadConfig reads the Telegram config and rejects missing fields.
|
||||
func (s *Store) LoadConfig(ctx context.Context) (AppConfig, error) {
|
||||
raw, err := s.rdb.Get(ctx, configKey()).Result()
|
||||
if errors.Is(err, redis.Nil) {
|
||||
return AppConfig{}, errors.New("telegram config not found in Redis. run configure first")
|
||||
}
|
||||
if err != nil {
|
||||
return AppConfig{}, fmt.Errorf("read telegram config: %w", err)
|
||||
}
|
||||
|
||||
var cfg AppConfig
|
||||
if err := json.Unmarshal([]byte(raw), &cfg); err != nil {
|
||||
return AppConfig{}, errors.New("telegram config in Redis is not valid JSON")
|
||||
}
|
||||
|
||||
var missing []string
|
||||
if cfg.APIID == 0 {
|
||||
missing = append(missing, "api_id")
|
||||
}
|
||||
if cfg.APIHash == "" {
|
||||
missing = append(missing, "api_hash")
|
||||
}
|
||||
if cfg.Phone == "" {
|
||||
missing = append(missing, "phone")
|
||||
}
|
||||
if len(cfg.GroupIDs) == 0 {
|
||||
missing = append(missing, "group_ids")
|
||||
}
|
||||
if len(missing) > 0 {
|
||||
return AppConfig{}, fmt.Errorf("telegram config missing: %s", strings.Join(missing, ", "))
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
// SaveConfig replaces the Telegram config.
|
||||
func (s *Store) SaveConfig(ctx context.Context, cfg AppConfig) error {
|
||||
raw, err := json.Marshal(cfg)
|
||||
if err != nil {
|
||||
return fmt.Errorf("encode telegram config: %w", err)
|
||||
}
|
||||
if err := s.rdb.Set(ctx, configKey(), raw, 0).Err(); err != nil {
|
||||
return fmt.Errorf("save telegram config: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestConfigRoundTrip(t *testing.T) {
|
||||
st, _ := newTestStore(t)
|
||||
ctx := context.Background()
|
||||
want := AppConfig{APIID: 123, APIHash: "hash", Phone: "+15551234567", GroupIDs: []int64{-1001, -2}}
|
||||
|
||||
if err := st.SaveConfig(ctx, want); err != nil {
|
||||
t.Fatalf("SaveConfig: %v", err)
|
||||
}
|
||||
got, err := st.LoadConfig(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadConfig: %v", err)
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("LoadConfig = %+v, want %+v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadConfigReadsPythonWrittenJSON(t *testing.T) {
|
||||
st, mr := newTestStore(t)
|
||||
mr.Set("telegram-export:config",
|
||||
`{"api_id": 123, "api_hash": "hash", "phone": "+1555", "group_ids": [-1001234567890]}`)
|
||||
|
||||
got, err := st.LoadConfig(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("LoadConfig: %v", err)
|
||||
}
|
||||
if got.APIID != 123 || got.GroupIDs[0] != -1001234567890 {
|
||||
t.Fatalf("LoadConfig = %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadConfigErrors(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
value string
|
||||
want string
|
||||
}{
|
||||
{"missing key", "", "run configure first"},
|
||||
{"corrupt json", "{", "not valid JSON"},
|
||||
{"missing fields", `{"api_id": 1}`, "missing: api_hash, phone, group_ids"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
st, mr := newTestStore(t)
|
||||
if tt.value != "" {
|
||||
mr.Set("telegram-export:config", tt.value)
|
||||
}
|
||||
_, err := st.LoadConfig(context.Background())
|
||||
if err == nil || !strings.Contains(err.Error(), tt.want) {
|
||||
t.Fatalf("LoadConfig error = %v, want it to contain %q", err, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
// RunTimeLayout is the yyyymmddhhmmss crawl timestamp shared by every group
|
||||
// in one run. Fixed width, so timestamps sort chronologically as strings.
|
||||
const RunTimeLayout = "20060102150405"
|
||||
|
||||
// Member is one group member as stored in an export.
|
||||
type Member struct {
|
||||
ID int64 `json:"id"`
|
||||
Username string `json:"username"`
|
||||
FirstName string `json:"first_name"`
|
||||
LastName string `json:"last_name"`
|
||||
}
|
||||
|
||||
// GroupExport is one group's members for one run. Each export is a single
|
||||
// self-contained key, so deleting any key never corrupts another.
|
||||
type GroupExport struct {
|
||||
GroupID int64 `json:"group_id"`
|
||||
Title string `json:"title"`
|
||||
Time string `json:"time"`
|
||||
Members []Member `json:"members"`
|
||||
}
|
||||
|
||||
// NewRunTime returns the current time as a run timestamp.
|
||||
func NewRunTime() string {
|
||||
return time.Now().Format(RunTimeLayout)
|
||||
}
|
||||
|
||||
// IsRunTime reports whether value has the yyyymmddhhmmss shape.
|
||||
func IsRunTime(value string) bool {
|
||||
if len(value) != len(RunTimeLayout) {
|
||||
return false
|
||||
}
|
||||
for _, r := range value {
|
||||
if r < '0' || r > '9' {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func groupExportKey(groupID int64, runTime string) string {
|
||||
return key("group", strconv.FormatInt(groupID, 10), runTime)
|
||||
}
|
||||
|
||||
// SaveGroupExport writes one group's export with a single atomic SET.
|
||||
func (s *Store) SaveGroupExport(ctx context.Context, record GroupExport) error {
|
||||
raw, err := json.Marshal(record)
|
||||
if err != nil {
|
||||
return fmt.Errorf("encode export: %w", err)
|
||||
}
|
||||
if err := s.rdb.Set(ctx, groupExportKey(record.GroupID, record.Time), raw, 0).Err(); err != nil {
|
||||
return fmt.Errorf("save export for group %d: %w", record.GroupID, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// GroupExportTimes returns one group's run timestamps, oldest first.
|
||||
func (s *Store) GroupExportTimes(ctx context.Context, groupID int64) ([]string, error) {
|
||||
pattern := groupExportKey(groupID, "*")
|
||||
prefix := strings.TrimSuffix(pattern, "*")
|
||||
|
||||
var times []string
|
||||
iter := s.rdb.Scan(ctx, 0, pattern, 100).Iterator()
|
||||
for iter.Next(ctx) {
|
||||
runTime := strings.TrimPrefix(iter.Val(), prefix)
|
||||
if IsRunTime(runTime) {
|
||||
times = append(times, runTime)
|
||||
}
|
||||
}
|
||||
if err := iter.Err(); err != nil {
|
||||
return nil, fmt.Errorf("list exports for group %d: %w", groupID, err)
|
||||
}
|
||||
sort.Strings(times)
|
||||
return times, nil
|
||||
}
|
||||
|
||||
// ErrExportNotFound means no readable export exists at that group and time.
|
||||
var ErrExportNotFound = errors.New("export not found")
|
||||
|
||||
// GetGroupExport returns one export, or ErrExportNotFound when it is missing
|
||||
// or not valid JSON.
|
||||
func (s *Store) GetGroupExport(ctx context.Context, groupID int64, runTime string) (GroupExport, error) {
|
||||
raw, err := s.rdb.Get(ctx, groupExportKey(groupID, runTime)).Bytes()
|
||||
if errors.Is(err, redis.Nil) {
|
||||
return GroupExport{}, ErrExportNotFound
|
||||
}
|
||||
if err != nil {
|
||||
return GroupExport{}, fmt.Errorf("read export for group %d at %s: %w", groupID, runTime, err)
|
||||
}
|
||||
|
||||
var record GroupExport
|
||||
if err := json.Unmarshal(raw, &record); err != nil {
|
||||
return GroupExport{}, ErrExportNotFound
|
||||
}
|
||||
return record, nil
|
||||
}
|
||||
|
||||
// LatestGroupExports returns up to n readable exports for a group, oldest
|
||||
// first. Corrupt values and keys deleted mid-scan are skipped.
|
||||
func (s *Store) LatestGroupExports(ctx context.Context, groupID int64, n int) ([]GroupExport, error) {
|
||||
times, err := s.GroupExportTimes(ctx, groupID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var records []GroupExport
|
||||
for i := len(times) - 1; i >= 0 && len(records) < n; i-- {
|
||||
record, err := s.GetGroupExport(ctx, groupID, times[i])
|
||||
if errors.Is(err, ErrExportNotFound) {
|
||||
continue
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
records = append([]GroupExport{record}, records...)
|
||||
}
|
||||
return records, nil
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func saveExport(t *testing.T, st *Store, groupID int64, runTime string) {
|
||||
t.Helper()
|
||||
err := st.SaveGroupExport(context.Background(), GroupExport{
|
||||
GroupID: groupID,
|
||||
Title: "Group",
|
||||
Time: runTime,
|
||||
Members: []Member{{ID: 1, Username: "amy"}},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("SaveGroupExport: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGroupExportRoundTrip(t *testing.T) {
|
||||
st, _ := newTestStore(t)
|
||||
saveExport(t, st, -100, "20260724120000")
|
||||
|
||||
got, err := st.GetGroupExport(context.Background(), -100, "20260724120000")
|
||||
if err != nil {
|
||||
t.Fatalf("GetGroupExport: %v", err)
|
||||
}
|
||||
if got.Title != "Group" || len(got.Members) != 1 || got.Members[0].Username != "amy" {
|
||||
t.Fatalf("GetGroupExport = %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetGroupExportMissingOrCorrupt(t *testing.T) {
|
||||
st, mr := newTestStore(t)
|
||||
mr.Set("telegram-export:group:-100:20260724120000", "not json")
|
||||
|
||||
for _, runTime := range []string{"20260724120000", "20990101000000"} {
|
||||
_, err := st.GetGroupExport(context.Background(), -100, runTime)
|
||||
if !errors.Is(err, ErrExportNotFound) {
|
||||
t.Fatalf("GetGroupExport(%s) error = %v, want ErrExportNotFound", runTime, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestGroupExportTimesScopesToGroupAndSorts(t *testing.T) {
|
||||
st, mr := newTestStore(t)
|
||||
saveExport(t, st, -100, "20260724120000")
|
||||
saveExport(t, st, -100, "20260701000000")
|
||||
saveExport(t, st, -1000, "20260801000000") // prefix of another group's ID
|
||||
mr.Set("telegram-export:group:-100:latest", "{}")
|
||||
|
||||
got, err := st.GroupExportTimes(context.Background(), -100)
|
||||
if err != nil {
|
||||
t.Fatalf("GroupExportTimes: %v", err)
|
||||
}
|
||||
want := []string{"20260701000000", "20260724120000"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("GroupExportTimes = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLatestGroupExportsSkipsCorrupt(t *testing.T) {
|
||||
st, mr := newTestStore(t)
|
||||
saveExport(t, st, -100, "20260701000000")
|
||||
saveExport(t, st, -100, "20260702000000")
|
||||
mr.Set("telegram-export:group:-100:20260703000000", "{broken")
|
||||
|
||||
got, err := st.LatestGroupExports(context.Background(), -100, 2)
|
||||
if err != nil {
|
||||
t.Fatalf("LatestGroupExports: %v", err)
|
||||
}
|
||||
if len(got) != 2 || got[0].Time != "20260701000000" || got[1].Time != "20260702000000" {
|
||||
t.Fatalf("LatestGroupExports = %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsRunTime(t *testing.T) {
|
||||
for value, want := range map[string]bool{
|
||||
"20260724120000": true,
|
||||
"2026072412000": false,
|
||||
"2026072412000a": false,
|
||||
"202607241200000": false,
|
||||
} {
|
||||
if got := IsRunTime(value); got != want {
|
||||
t.Errorf("IsRunTime(%q) = %v, want %v", value, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
// Package store owns every Redis key the tool reads or writes. Other packages
|
||||
// never import go-redis directly.
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/joho/godotenv"
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
// keyPrefix namespaces every key. It is fixed to keep local setup to one
|
||||
// REDIS_URL value.
|
||||
const keyPrefix = "telegram-export"
|
||||
|
||||
// Store wraps the shared Redis that holds config, session, and crawl history,
|
||||
// so the tool runs on any device from just a .env file.
|
||||
type Store struct {
|
||||
rdb *redis.Client
|
||||
}
|
||||
|
||||
// OpenFromEnv loads .env, then connects to REDIS_URL.
|
||||
func OpenFromEnv(ctx context.Context) (*Store, error) {
|
||||
_ = godotenv.Load()
|
||||
|
||||
redisURL := os.Getenv("REDIS_URL")
|
||||
if redisURL == "" {
|
||||
return nil, errors.New("REDIS_URL not set in .env (e.g. rediss://default:<password>@<host>:<port>)")
|
||||
}
|
||||
return Open(ctx, redisURL)
|
||||
}
|
||||
|
||||
// Open connects to Redis and checks the connection.
|
||||
func Open(ctx context.Context, redisURL string) (*Store, error) {
|
||||
opts, err := redis.ParseURL(redisURL)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("parse REDIS_URL: %w", err)
|
||||
}
|
||||
rdb := redis.NewClient(opts)
|
||||
if err := rdb.Ping(ctx).Err(); err != nil {
|
||||
_ = rdb.Close()
|
||||
return nil, fmt.Errorf("connect to Redis: %w", err)
|
||||
}
|
||||
return &Store{rdb: rdb}, nil
|
||||
}
|
||||
|
||||
// Close releases the Redis connection.
|
||||
func (s *Store) Close() error {
|
||||
return s.rdb.Close()
|
||||
}
|
||||
|
||||
// key builds a namespaced Redis key; the single owner of the prefix.
|
||||
func key(parts ...string) string {
|
||||
return strings.Join(append([]string{keyPrefix}, parts...), ":")
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"github.com/gotd/td/session"
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
// The login session grants full account access; it lives only in Redis
|
||||
// (behind TLS/auth), never on local disk. Deleting it forces a clean re-login.
|
||||
//
|
||||
// The Python version stored a Telethon StringSession under "session". This
|
||||
// version keeps its own format under a separate key and only reads the old
|
||||
// one once, so an existing login carries over without a new code.
|
||||
func sessionKey() string { return key("session-gotd") }
|
||||
func telethonSessionKey() string { return key("session") }
|
||||
|
||||
// SessionStorage implements gotd's session.Storage on Redis.
|
||||
type SessionStorage struct {
|
||||
store *Store
|
||||
}
|
||||
|
||||
// SessionStorage returns the Redis-backed Telegram session storage.
|
||||
func (s *Store) SessionStorage() *SessionStorage {
|
||||
return &SessionStorage{store: s}
|
||||
}
|
||||
|
||||
// LoadSession returns the stored session, importing a Telethon session when
|
||||
// no gotd session exists yet.
|
||||
func (ss *SessionStorage) LoadSession(ctx context.Context) ([]byte, error) {
|
||||
raw, err := ss.store.rdb.Get(ctx, sessionKey()).Bytes()
|
||||
if err == nil {
|
||||
return raw, nil
|
||||
}
|
||||
if !errors.Is(err, redis.Nil) {
|
||||
return nil, fmt.Errorf("read session: %w", err)
|
||||
}
|
||||
|
||||
telethon, err := ss.store.rdb.Get(ctx, telethonSessionKey()).Result()
|
||||
if errors.Is(err, redis.Nil) || telethon == "" {
|
||||
return nil, session.ErrNotFound
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read Telethon session: %w", err)
|
||||
}
|
||||
|
||||
raw, err = convertTelethonSession(ctx, telethon)
|
||||
if err != nil {
|
||||
// An unreadable old session should mean a fresh login, not a dead end.
|
||||
return nil, session.ErrNotFound
|
||||
}
|
||||
if err := ss.StoreSession(ctx, raw); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return raw, nil
|
||||
}
|
||||
|
||||
// StoreSession persists the session for reuse on any device.
|
||||
func (ss *SessionStorage) StoreSession(ctx context.Context, data []byte) error {
|
||||
if err := ss.store.rdb.Set(ctx, sessionKey(), data, 0).Err(); err != nil {
|
||||
return fmt.Errorf("save session: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func convertTelethonSession(ctx context.Context, telethon string) ([]byte, error) {
|
||||
data, err := session.TelethonSession(telethon)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var mem session.StorageMemory
|
||||
if err := (&session.Loader{Storage: &mem}).Save(ctx, data); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return mem.LoadSession(ctx)
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/gotd/td/session"
|
||||
)
|
||||
|
||||
// telethonString builds a Telethon StringSession: version "1" plus base64
|
||||
// of DC ID, IPv4, port, and a 256-byte auth key.
|
||||
func telethonString(dc byte) string {
|
||||
raw := []byte{dc, 149, 154, 167, 51, 0x01, 0xbb}
|
||||
raw = append(raw, bytes.Repeat([]byte{7}, 256)...)
|
||||
return "1" + base64.URLEncoding.EncodeToString(raw)
|
||||
}
|
||||
|
||||
func TestLoadSessionNotFound(t *testing.T) {
|
||||
st, _ := newTestStore(t)
|
||||
|
||||
_, err := st.SessionStorage().LoadSession(context.Background())
|
||||
if !errors.Is(err, session.ErrNotFound) {
|
||||
t.Fatalf("LoadSession error = %v, want session.ErrNotFound", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionRoundTrip(t *testing.T) {
|
||||
st, _ := newTestStore(t)
|
||||
ctx := context.Background()
|
||||
|
||||
if err := st.SessionStorage().StoreSession(ctx, []byte(`{"Version":1}`)); err != nil {
|
||||
t.Fatalf("StoreSession: %v", err)
|
||||
}
|
||||
got, err := st.SessionStorage().LoadSession(ctx)
|
||||
if err != nil || string(got) != `{"Version":1}` {
|
||||
t.Fatalf("LoadSession = %q, %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadSessionImportsTelethonSession(t *testing.T) {
|
||||
st, mr := newTestStore(t)
|
||||
ctx := context.Background()
|
||||
mr.Set("telegram-export:session", telethonString(2))
|
||||
|
||||
raw, err := st.SessionStorage().LoadSession(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadSession: %v", err)
|
||||
}
|
||||
data, err := (&session.Loader{Storage: st.SessionStorage()}).Load(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("Loader.Load: %v", err)
|
||||
}
|
||||
if data.DC != 2 || data.Addr != "149.154.167.51:443" {
|
||||
t.Fatalf("imported session DC=%d Addr=%q", data.DC, data.Addr)
|
||||
}
|
||||
if stored, _ := mr.Get("telegram-export:session-gotd"); stored != string(raw) {
|
||||
t.Fatal("imported session was not saved under the gotd key")
|
||||
}
|
||||
if old, _ := mr.Get("telegram-export:session"); old != telethonString(2) {
|
||||
t.Fatal("Telethon session key was modified")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadSessionIgnoresBrokenTelethonSession(t *testing.T) {
|
||||
st, mr := newTestStore(t)
|
||||
mr.Set("telegram-export:session", "not-a-session")
|
||||
|
||||
_, err := st.SessionStorage().LoadSession(context.Background())
|
||||
if !errors.Is(err, session.ErrNotFound) {
|
||||
t.Fatalf("LoadSession error = %v, want session.ErrNotFound", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/alicebob/miniredis/v2"
|
||||
)
|
||||
|
||||
func newTestStore(t *testing.T) (*Store, *miniredis.Miniredis) {
|
||||
t.Helper()
|
||||
mr := miniredis.RunT(t)
|
||||
st, err := Open(context.Background(), "redis://"+mr.Addr())
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = st.Close() })
|
||||
return st, mr
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
// Package telegram logs in to Telegram and reads group members.
|
||||
package telegram
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/gotd/td/session"
|
||||
"github.com/gotd/td/telegram"
|
||||
"github.com/gotd/td/telegram/auth"
|
||||
"github.com/gotd/td/tg"
|
||||
"github.com/gotd/td/tgerr"
|
||||
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/prompt"
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/store"
|
||||
)
|
||||
|
||||
// Run connects with the stored session, signs in only when that session is
|
||||
// missing or expired, and calls fn while the connection is open.
|
||||
func Run(
|
||||
ctx context.Context,
|
||||
cfg store.AppConfig,
|
||||
storage session.Storage,
|
||||
ask *prompt.Prompter,
|
||||
fn func(ctx context.Context, api *tg.Client) error,
|
||||
) error {
|
||||
client := telegram.NewClient(cfg.APIID, cfg.APIHash, telegram.Options{
|
||||
SessionStorage: storage,
|
||||
NoUpdates: true,
|
||||
})
|
||||
return client.Run(ctx, func(ctx context.Context) error {
|
||||
flow := auth.NewFlow(terminalAuth{phone: cfg.Phone, ask: ask}, auth.SendCodeOptions{})
|
||||
if err := client.Auth().IfNecessary(ctx, flow); err != nil {
|
||||
return fmt.Errorf("telegram login: %w", err)
|
||||
}
|
||||
return fn(ctx, client.API())
|
||||
})
|
||||
}
|
||||
|
||||
// terminalAuth answers the login flow from the terminal.
|
||||
type terminalAuth struct {
|
||||
phone string
|
||||
ask *prompt.Prompter
|
||||
}
|
||||
|
||||
func (a terminalAuth) Phone(context.Context) (string, error) { return a.phone, nil }
|
||||
|
||||
func (a terminalAuth) Password(context.Context) (string, error) {
|
||||
return a.ask.Secret("Password")
|
||||
}
|
||||
|
||||
func (a terminalAuth) Code(context.Context, *tg.AuthSentCode) (string, error) {
|
||||
return a.ask.Line("Enter code")
|
||||
}
|
||||
|
||||
func (a terminalAuth) AcceptTermsOfService(context.Context, tg.HelpTermsOfService) error {
|
||||
return errors.New("this phone has no Telegram account; sign up in an official app first")
|
||||
}
|
||||
|
||||
func (a terminalAuth) SignUp(context.Context) (auth.UserInfo, error) {
|
||||
return auth.UserInfo{}, errors.New("this phone has no Telegram account; sign up in an official app first")
|
||||
}
|
||||
|
||||
// maxFloodWait caps how long one call may sleep on FLOOD_WAIT before giving up.
|
||||
const maxFloodWait = 5 * time.Minute
|
||||
|
||||
// withFloodWait retries call after Telegram's requested FLOOD_WAIT delay, so
|
||||
// paging a large group does not fail on rate limits.
|
||||
func withFloodWait[T any](ctx context.Context, call func() (T, error)) (T, error) {
|
||||
for {
|
||||
result, err := call()
|
||||
wait, ok := tgerr.AsFloodWait(err)
|
||||
if !ok || wait > maxFloodWait {
|
||||
return result, err
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return result, ctx.Err()
|
||||
case <-time.After(wait + time.Second):
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
package telegram
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/gotd/td/telegram/query/dialogs"
|
||||
"github.com/gotd/td/tg"
|
||||
)
|
||||
|
||||
// dialogsPageSize is how many dialogs each lookup request fetches.
|
||||
const dialogsPageSize = 100
|
||||
|
||||
// Fetcher resolves group IDs and fetches their members.
|
||||
//
|
||||
// A marked group ID alone cannot address a supergroup: Telegram also needs
|
||||
// its access hash. The account's dialog list carries both, and the account
|
||||
// must already be in the group to see its members, so the list is read once
|
||||
// and reused for every group in a run.
|
||||
type Fetcher struct {
|
||||
api *tg.Client
|
||||
groups map[groupRef]resolvedGroup
|
||||
}
|
||||
|
||||
type resolvedGroup struct {
|
||||
title string
|
||||
channel *tg.InputChannel
|
||||
}
|
||||
|
||||
// NewFetcher returns a Fetcher on an open connection.
|
||||
func NewFetcher(api *tg.Client) *Fetcher {
|
||||
return &Fetcher{api: api}
|
||||
}
|
||||
|
||||
// FetchGroup returns the group's title and members. It returns ErrNotGroup
|
||||
// for a user ID.
|
||||
func (f *Fetcher) FetchGroup(ctx context.Context, markedID int64, onStart func(title string)) (Group, error) {
|
||||
ref, err := parseGroupID(markedID)
|
||||
if err != nil {
|
||||
return Group{}, err
|
||||
}
|
||||
if err := f.loadDialogs(ctx); err != nil {
|
||||
return Group{}, err
|
||||
}
|
||||
resolved, ok := f.groups[ref]
|
||||
if !ok {
|
||||
return Group{}, fmt.Errorf("group %d not found in this account's chats", markedID)
|
||||
}
|
||||
|
||||
onStart(resolved.title)
|
||||
group := Group{ID: markedID, Title: resolved.title}
|
||||
if ref.kind == channelGroup {
|
||||
group.Members, err = fetchChannelMembers(ctx, f.api, resolved.channel)
|
||||
} else {
|
||||
group.Members, err = fetchBasicGroupMembers(ctx, f.api, ref.id)
|
||||
}
|
||||
if err != nil {
|
||||
return Group{}, err
|
||||
}
|
||||
return group, nil
|
||||
}
|
||||
|
||||
func (f *Fetcher) loadDialogs(ctx context.Context) error {
|
||||
if f.groups != nil {
|
||||
return nil
|
||||
}
|
||||
groups := map[groupRef]resolvedGroup{}
|
||||
iter := dialogs.NewQueryBuilder(f.api).GetDialogs().BatchSize(dialogsPageSize).Iter()
|
||||
for iter.Next(ctx) {
|
||||
elem := iter.Value()
|
||||
switch peer := elem.Peer.(type) {
|
||||
case *tg.InputPeerChat:
|
||||
if chat, ok := elem.Entities.Chat(peer.ChatID); ok {
|
||||
groups[groupRef{kind: basicGroup, id: peer.ChatID}] = resolvedGroup{title: chat.Title}
|
||||
}
|
||||
case *tg.InputPeerChannel:
|
||||
if channel, ok := elem.Entities.Channel(peer.ChannelID); ok {
|
||||
groups[groupRef{kind: channelGroup, id: peer.ChannelID}] = resolvedGroup{
|
||||
title: channel.Title,
|
||||
channel: &tg.InputChannel{ChannelID: peer.ChannelID, AccessHash: peer.AccessHash},
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := iter.Err(); err != nil {
|
||||
return fmt.Errorf("list chats: %w", err)
|
||||
}
|
||||
f.groups = groups
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
package telegram
|
||||
|
||||
import "errors"
|
||||
|
||||
// ErrNotGroup means the ID belongs to a user, not a group or channel.
|
||||
var ErrNotGroup = errors.New("not a group")
|
||||
|
||||
// channelIDOffset is how Bot API style "marked" IDs encode a channel or
|
||||
// supergroup: -100<channel_id>, i.e. -(1e12 + channel_id).
|
||||
const channelIDOffset = 1_000_000_000_000
|
||||
|
||||
type groupKind int
|
||||
|
||||
const (
|
||||
basicGroup groupKind = iota
|
||||
channelGroup
|
||||
)
|
||||
|
||||
// groupRef is a parsed marked group ID.
|
||||
type groupRef struct {
|
||||
kind groupKind
|
||||
id int64
|
||||
}
|
||||
|
||||
// parseGroupID splits a marked ID (as Telethon and the Python version store
|
||||
// it) into a basic group or a channel/supergroup and its raw Telegram ID.
|
||||
func parseGroupID(markedID int64) (groupRef, error) {
|
||||
switch {
|
||||
case markedID <= -channelIDOffset:
|
||||
return groupRef{kind: channelGroup, id: -markedID - channelIDOffset}, nil
|
||||
case markedID < 0:
|
||||
return groupRef{kind: basicGroup, id: -markedID}, nil
|
||||
default:
|
||||
return groupRef{}, ErrNotGroup
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
package telegram
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/gotd/td/tg"
|
||||
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/store"
|
||||
)
|
||||
|
||||
func TestParseGroupID(t *testing.T) {
|
||||
tests := []struct {
|
||||
marked int64
|
||||
want groupRef
|
||||
}{
|
||||
{-1001234567890, groupRef{kind: channelGroup, id: 1234567890}},
|
||||
{-1000000000001, groupRef{kind: channelGroup, id: 1}},
|
||||
{-123456, groupRef{kind: basicGroup, id: 123456}},
|
||||
{-999999999999, groupRef{kind: basicGroup, id: 999999999999}},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
got, err := parseGroupID(tt.marked)
|
||||
if err != nil || got != tt.want {
|
||||
t.Errorf("parseGroupID(%d) = %+v, %v; want %+v", tt.marked, got, err, tt.want)
|
||||
}
|
||||
}
|
||||
|
||||
for _, userID := range []int64{0, 777} {
|
||||
if _, err := parseGroupID(userID); !errors.Is(err, ErrNotGroup) {
|
||||
t.Errorf("parseGroupID(%d) error = %v, want ErrNotGroup", userID, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestChannelParticipantUserID(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
participant tg.ChannelParticipantClass
|
||||
want int64
|
||||
ok bool
|
||||
}{
|
||||
{"member", &tg.ChannelParticipant{UserID: 1}, 1, true},
|
||||
{"self", &tg.ChannelParticipantSelf{UserID: 2}, 2, true},
|
||||
{"admin", &tg.ChannelParticipantAdmin{UserID: 3, InviterID: 99}, 3, true},
|
||||
{"creator", &tg.ChannelParticipantCreator{UserID: 4}, 4, true},
|
||||
{"banned user", &tg.ChannelParticipantBanned{Peer: &tg.PeerUser{UserID: 5}}, 5, true},
|
||||
{"banned channel", &tg.ChannelParticipantBanned{Peer: &tg.PeerChannel{ChannelID: 6}}, 0, false},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
got, ok := channelParticipantUserID(tt.participant)
|
||||
if got != tt.want || ok != tt.ok {
|
||||
t.Errorf("%s: got %d, %v; want %d, %v", tt.name, got, ok, tt.want, tt.ok)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMemberFromUserAndSorting(t *testing.T) {
|
||||
users := usersByID([]tg.UserClass{
|
||||
&tg.User{ID: 9, Username: "zed", FirstName: "Z"},
|
||||
&tg.User{ID: 3, FirstName: "C", LastName: "D"},
|
||||
&tg.UserEmpty{ID: 5},
|
||||
})
|
||||
if len(users) != 2 {
|
||||
t.Fatalf("usersByID kept %d users, want 2", len(users))
|
||||
}
|
||||
|
||||
got := sortedMembers(map[int64]store.Member{
|
||||
9: memberFromUser(users[9]),
|
||||
3: memberFromUser(users[3]),
|
||||
})
|
||||
if got[0].ID != 3 || got[0].LastName != "D" || got[1].Username != "zed" {
|
||||
t.Fatalf("sortedMembers = %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
package telegram
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
|
||||
"github.com/gotd/td/tg"
|
||||
|
||||
"github.com/tiennm99/export-telegram-group-members/internal/store"
|
||||
)
|
||||
|
||||
// participantsPageSize is the largest page channels.getParticipants allows.
|
||||
const participantsPageSize = 200
|
||||
|
||||
// Group is one fetched group with its members sorted by ID.
|
||||
type Group struct {
|
||||
ID int64
|
||||
Title string
|
||||
Members []store.Member
|
||||
}
|
||||
|
||||
// fetchChannelMembers pages through a channel or supergroup's members.
|
||||
func fetchChannelMembers(ctx context.Context, api *tg.Client, channel *tg.InputChannel) ([]store.Member, error) {
|
||||
byID := map[int64]store.Member{}
|
||||
for offset := 0; ; {
|
||||
resp, err := withFloodWait(ctx, func() (tg.ChannelsChannelParticipantsClass, error) {
|
||||
return api.ChannelsGetParticipants(ctx, &tg.ChannelsGetParticipantsRequest{
|
||||
Channel: channel,
|
||||
Filter: &tg.ChannelParticipantsSearch{},
|
||||
Offset: offset,
|
||||
Limit: participantsPageSize,
|
||||
})
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("get participants: %w", err)
|
||||
}
|
||||
page, ok := resp.(*tg.ChannelsChannelParticipants)
|
||||
if !ok || len(page.Participants) == 0 {
|
||||
break
|
||||
}
|
||||
users := usersByID(page.Users)
|
||||
for _, participant := range page.Participants {
|
||||
if userID, ok := channelParticipantUserID(participant); ok {
|
||||
if user, ok := users[userID]; ok {
|
||||
byID[userID] = memberFromUser(user)
|
||||
}
|
||||
}
|
||||
}
|
||||
offset += len(page.Participants)
|
||||
}
|
||||
return sortedMembers(byID), nil
|
||||
}
|
||||
|
||||
// fetchBasicGroupMembers reads a basic group's members from its full info.
|
||||
func fetchBasicGroupMembers(ctx context.Context, api *tg.Client, chatID int64) ([]store.Member, error) {
|
||||
full, err := withFloodWait(ctx, func() (*tg.MessagesChatFull, error) {
|
||||
return api.MessagesGetFullChat(ctx, chatID)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("get full chat: %w", err)
|
||||
}
|
||||
chatFull, ok := full.FullChat.(*tg.ChatFull)
|
||||
if !ok {
|
||||
return nil, errors.New("unexpected full chat type")
|
||||
}
|
||||
participants, ok := chatFull.Participants.(*tg.ChatParticipants)
|
||||
if !ok {
|
||||
return nil, errors.New("member list is hidden")
|
||||
}
|
||||
|
||||
users := usersByID(full.Users)
|
||||
byID := map[int64]store.Member{}
|
||||
for _, participant := range participants.Participants {
|
||||
if user, ok := users[participant.GetUserID()]; ok {
|
||||
byID[user.ID] = memberFromUser(user)
|
||||
}
|
||||
}
|
||||
return sortedMembers(byID), nil
|
||||
}
|
||||
|
||||
// channelParticipantUserID returns the member's user ID. Admin entries also
|
||||
// reference inviters and promoters, so users come from participants only.
|
||||
func channelParticipantUserID(participant tg.ChannelParticipantClass) (int64, bool) {
|
||||
switch p := participant.(type) {
|
||||
case *tg.ChannelParticipant:
|
||||
return p.UserID, true
|
||||
case *tg.ChannelParticipantSelf:
|
||||
return p.UserID, true
|
||||
case *tg.ChannelParticipantAdmin:
|
||||
return p.UserID, true
|
||||
case *tg.ChannelParticipantCreator:
|
||||
return p.UserID, true
|
||||
case *tg.ChannelParticipantBanned:
|
||||
if peer, ok := p.Peer.(*tg.PeerUser); ok {
|
||||
return peer.UserID, true
|
||||
}
|
||||
case *tg.ChannelParticipantLeft:
|
||||
if peer, ok := p.Peer.(*tg.PeerUser); ok {
|
||||
return peer.UserID, true
|
||||
}
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
func usersByID(users []tg.UserClass) map[int64]*tg.User {
|
||||
byID := make(map[int64]*tg.User, len(users))
|
||||
for _, u := range users {
|
||||
if user, ok := u.(*tg.User); ok {
|
||||
byID[user.ID] = user
|
||||
}
|
||||
}
|
||||
return byID
|
||||
}
|
||||
|
||||
func memberFromUser(user *tg.User) store.Member {
|
||||
return store.Member{
|
||||
ID: user.ID,
|
||||
Username: user.Username,
|
||||
FirstName: user.FirstName,
|
||||
LastName: user.LastName,
|
||||
}
|
||||
}
|
||||
|
||||
func sortedMembers(byID map[int64]store.Member) []store.Member {
|
||||
list := make([]store.Member, 0, len(byID))
|
||||
for _, member := range byID {
|
||||
list = append(list, member)
|
||||
}
|
||||
sort.Slice(list, func(i, j int) bool { return list[i].ID < list[j].ID })
|
||||
return list
|
||||
}
|
||||
@@ -1,165 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/celestix/gotgproto"
|
||||
"github.com/celestix/gotgproto/ext"
|
||||
"github.com/celestix/gotgproto/sessionMaker"
|
||||
"github.com/gotd/td/tg"
|
||||
"github.com/joho/godotenv"
|
||||
"gorm.io/driver/sqlite"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if err := godotenv.Load(); err != nil {
|
||||
log.Println("Warning: .env file not found")
|
||||
}
|
||||
phoneNumber, isExist := os.LookupEnv("TG_PHONE")
|
||||
if !isExist {
|
||||
log.Fatal("TG_PHONE not set")
|
||||
}
|
||||
rawAppId, isExist := os.LookupEnv("APP_ID")
|
||||
if !isExist {
|
||||
log.Fatal("APP_ID not set!")
|
||||
return
|
||||
}
|
||||
appId, err := strconv.Atoi(rawAppId)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
return
|
||||
}
|
||||
appHash, isExist := os.LookupEnv("APP_HASH")
|
||||
if !isExist {
|
||||
log.Fatal("APP_HASH not set")
|
||||
return
|
||||
}
|
||||
rawGroupIds, isExist := os.LookupEnv("GROUP_IDS")
|
||||
if !isExist {
|
||||
log.Fatal("GROUP_IDS not set")
|
||||
return
|
||||
}
|
||||
groupIds, err := strToInts(rawGroupIds)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
return
|
||||
}
|
||||
|
||||
client, err := gotgproto.NewClient(
|
||||
// Get AppID from https://my.telegram.org/apps
|
||||
appId,
|
||||
// Get ApiHash from https://my.telegram.org/apps
|
||||
appHash,
|
||||
// ClientType, as we defined above
|
||||
gotgproto.ClientTypePhone(phoneNumber),
|
||||
// Optional parameters of client
|
||||
&gotgproto.ClientOpts{
|
||||
Session: sessionMaker.SqlSession(sqlite.Open("session.sqlite3")),
|
||||
},
|
||||
)
|
||||
if err != nil {
|
||||
log.Fatalln("failed to start client:", err)
|
||||
}
|
||||
|
||||
fmt.Printf("client (@%s) has been started...\n", client.Self.Username)
|
||||
|
||||
ctx := client.CreateContext()
|
||||
api := client.API()
|
||||
|
||||
// Example: work with the first group ID
|
||||
groupID := groupIds[0]
|
||||
|
||||
inputPeer, err := ctx.ResolveInputPeerById(groupID)
|
||||
if err != nil {
|
||||
log.Fatalf("failed to resolve input peer: %v", err)
|
||||
}
|
||||
|
||||
p := inputPeer.(*tg.InputPeerChannel)
|
||||
|
||||
chat, err := ctx.GetChat(groupID)
|
||||
if err != nil {
|
||||
log.Fatalf("failed to get chat: %v", err)
|
||||
}
|
||||
|
||||
// Prepare input channel
|
||||
inputChannel := &tg.InputChannel{
|
||||
ChannelID: chat.GetID(),
|
||||
AccessHash: p.AccessHash,
|
||||
}
|
||||
|
||||
// Collect unique user IDs
|
||||
users := make(map[int64]struct{})
|
||||
|
||||
offset := 0
|
||||
limit := 100
|
||||
totalCount := 200
|
||||
|
||||
for offset < totalCount {
|
||||
resp, err := api.ChannelsGetParticipants(context.Background(), &tg.ChannelsGetParticipantsRequest{
|
||||
Channel: inputChannel,
|
||||
Filter: &tg.ChannelParticipantsRecent{},
|
||||
Offset: offset,
|
||||
Limit: limit,
|
||||
Hash: 0,
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatalf("failed to get participants: %v", err)
|
||||
}
|
||||
|
||||
data := resp.(*tg.ChannelsChannelParticipants)
|
||||
totalCount = data.Count
|
||||
|
||||
for _, p := range data.Participants {
|
||||
switch participant := p.(type) {
|
||||
case *tg.ChannelParticipantSelf:
|
||||
users[participant.UserID] = struct{}{}
|
||||
case *tg.ChannelParticipant:
|
||||
users[participant.UserID] = struct{}{}
|
||||
case *tg.ChannelParticipantAdmin:
|
||||
users[participant.UserID] = struct{}{}
|
||||
case *tg.ChannelParticipantCreator:
|
||||
users[participant.UserID] = struct{}{}
|
||||
}
|
||||
}
|
||||
|
||||
offset += limit
|
||||
log.Printf("Fetched %d / %d participants\n", offset, totalCount)
|
||||
}
|
||||
|
||||
log.Printf("Total unique participants: %d\n", len(users))
|
||||
|
||||
// Optional: convert map to slice
|
||||
userIDs := make([]int64, 0, len(users))
|
||||
for id := range users {
|
||||
userIDs = append(userIDs, id)
|
||||
}
|
||||
|
||||
log.Println("Done.")
|
||||
client.Idle()
|
||||
}
|
||||
|
||||
func echo(ctx *ext.Context, update *ext.Update) error {
|
||||
msg := update.EffectiveMessage
|
||||
_, err := ctx.Reply(update, ext.ReplyTextString(msg.Text), nil)
|
||||
return err
|
||||
}
|
||||
|
||||
func strToInts(s string) ([]int64, error) {
|
||||
parts := strings.Split(s, ",")
|
||||
ints := make([]int64, 0, len(parts))
|
||||
|
||||
for _, p := range parts {
|
||||
v, err := strconv.ParseInt(strings.TrimSpace(p), 10, 64)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ints = append(ints, v)
|
||||
}
|
||||
|
||||
return ints, nil
|
||||
}
|
||||
@@ -1,19 +0,0 @@
|
||||
"""Load/save the Telethon StringSession from Redis.
|
||||
|
||||
The session string grants full account access; it lives only in Redis (behind
|
||||
TLS/auth), never on local disk. Deleting the key simply forces a clean re-login.
|
||||
"""
|
||||
|
||||
from config import key, redis_client
|
||||
|
||||
_SESSION_KEY = key('session')
|
||||
|
||||
|
||||
def load_session():
|
||||
"""Return the stored StringSession string, or None if not yet authenticated."""
|
||||
return redis_client.get(_SESSION_KEY)
|
||||
|
||||
|
||||
def save_session(session_string):
|
||||
"""Persist the StringSession string for reuse on any device."""
|
||||
redis_client.set(_SESSION_KEY, session_string)
|
||||
@@ -1,40 +0,0 @@
|
||||
import os
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
os.environ.setdefault('REDIS_URL', 'redis://localhost')
|
||||
|
||||
import common
|
||||
|
||||
|
||||
class LatestGroupExportTimeTest(unittest.TestCase):
|
||||
def test_uses_latest_timestamp_from_redis_keys_without_fetching_records(self):
|
||||
redis_client = MagicMock()
|
||||
redis_client.scan_iter.return_value = [
|
||||
'telegram-export:group:-100123:20260724120000',
|
||||
'telegram-export:group:-100123:20260723120000',
|
||||
]
|
||||
|
||||
with patch.object(common, 'redis_client', redis_client):
|
||||
run_time = common.latest_group_export_time(-100123)
|
||||
|
||||
self.assertEqual(run_time, '20260724120000')
|
||||
redis_client.scan_iter.assert_called_once_with(
|
||||
match='telegram-export:group:-100123:*',
|
||||
)
|
||||
redis_client.get.assert_not_called()
|
||||
|
||||
def test_ignores_malformed_keys(self):
|
||||
redis_client = MagicMock()
|
||||
redis_client.scan_iter.return_value = [
|
||||
'telegram-export:group:-100123:not-a-time',
|
||||
]
|
||||
|
||||
with patch.object(common, 'redis_client', redis_client):
|
||||
run_time = common.latest_group_export_time(-100123)
|
||||
|
||||
self.assertIsNone(run_time)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -1,110 +0,0 @@
|
||||
import io
|
||||
import os
|
||||
import unittest
|
||||
from contextlib import redirect_stdout
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
os.environ.setdefault('REDIS_URL', 'redis://localhost')
|
||||
|
||||
import common
|
||||
import compare_groups
|
||||
import crawl
|
||||
|
||||
|
||||
class FakeChat:
|
||||
def __init__(self, group_id):
|
||||
self.id = group_id
|
||||
self.title = f'Group {group_id}'
|
||||
|
||||
|
||||
class FakeMember:
|
||||
def __init__(self, member_id, username=None):
|
||||
self.id = member_id
|
||||
self.username = username
|
||||
self.first_name = None
|
||||
self.last_name = None
|
||||
|
||||
|
||||
def group(group_id, member_ids):
|
||||
return {
|
||||
'group_id': group_id,
|
||||
'title': f'Group {group_id}',
|
||||
'members': [{'id': i, 'username': None, 'first_name': None, 'last_name': None}
|
||||
for i in member_ids],
|
||||
}
|
||||
|
||||
|
||||
class SplitMembersTest(unittest.TestCase):
|
||||
def test_splits_by_member_id(self):
|
||||
only_a, only_b, both = compare_groups.split_members(
|
||||
group(1, [3, 1, 2]), group(2, [4, 2, 3]),
|
||||
)
|
||||
|
||||
self.assertEqual([m['id'] for m in only_a], [1])
|
||||
self.assertEqual([m['id'] for m in only_b], [4])
|
||||
self.assertEqual([m['id'] for m in both], [2, 3])
|
||||
|
||||
|
||||
class CompareGroupsMainTest(unittest.TestCase):
|
||||
def test_requires_two_group_ids(self):
|
||||
with self.assertRaises(SystemExit):
|
||||
compare_groups.parse_args(['-100123'])
|
||||
|
||||
@patch.object(common, 'save_group_export')
|
||||
@patch.object(crawl, 'save_session')
|
||||
@patch.object(crawl, 'load_session', return_value='stored-session')
|
||||
@patch.object(crawl, 'TelegramClient')
|
||||
def test_fetches_live_and_does_not_save_exports(
|
||||
self, telegram_client, load_session, save_session, save_group_export,
|
||||
):
|
||||
client = MagicMock()
|
||||
telegram_client.return_value = client
|
||||
client.is_user_authorized.return_value = True
|
||||
client.get_entity.side_effect = [FakeChat(101), FakeChat(202)]
|
||||
client.get_participants.side_effect = [
|
||||
[FakeMember(2, 'bob'), FakeMember(1, 'amy')],
|
||||
[FakeMember(3, 'cat'), FakeMember(2, 'bob')],
|
||||
]
|
||||
app_config = {'api_id': 1, 'api_hash': 'h', 'phone': 'p', 'group_ids': [9]}
|
||||
|
||||
out = io.StringIO()
|
||||
with (
|
||||
patch('config.load_app_config', return_value=app_config),
|
||||
patch.object(crawl, 'Chat', FakeChat),
|
||||
patch.object(crawl, 'Channel', FakeChat),
|
||||
patch.object(crawl, 'StringSession', return_value='session'),
|
||||
redirect_stdout(out),
|
||||
):
|
||||
result = compare_groups.main(['101', '202'])
|
||||
|
||||
self.assertEqual(result, 0)
|
||||
save_group_export.assert_not_called()
|
||||
lines = out.getvalue().splitlines()
|
||||
self.assertIn(' 2 @bob', lines)
|
||||
self.assertIn('- 1 @amy', lines)
|
||||
self.assertIn('+ 3 @cat', lines)
|
||||
|
||||
@patch.object(crawl, 'save_session')
|
||||
@patch.object(crawl, 'load_session', return_value='stored-session')
|
||||
@patch.object(crawl, 'TelegramClient')
|
||||
def test_non_group_returns_error(self, telegram_client, load_session, save_session):
|
||||
client = MagicMock()
|
||||
telegram_client.return_value = client
|
||||
client.is_user_authorized.return_value = True
|
||||
client.get_entity.return_value = object()
|
||||
app_config = {'api_id': 1, 'api_hash': 'h', 'phone': 'p', 'group_ids': [9]}
|
||||
|
||||
with (
|
||||
patch('config.load_app_config', return_value=app_config),
|
||||
patch.object(crawl, 'StringSession', return_value='session'),
|
||||
redirect_stdout(io.StringIO()),
|
||||
patch('sys.stderr', io.StringIO()),
|
||||
):
|
||||
result = compare_groups.main(['101', '202'])
|
||||
|
||||
self.assertEqual(result, 1)
|
||||
client.get_participants.assert_not_called()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -1,132 +0,0 @@
|
||||
import os
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, call, patch
|
||||
|
||||
os.environ.setdefault('REDIS_URL', 'redis://localhost')
|
||||
|
||||
import crawl
|
||||
|
||||
|
||||
class FakeChat:
|
||||
def __init__(self, group_id):
|
||||
self.id = group_id
|
||||
self.title = f'Group {group_id}'
|
||||
|
||||
|
||||
class FakeMember:
|
||||
def __init__(self, member_id):
|
||||
self.id = member_id
|
||||
|
||||
|
||||
class CrawlArgumentsTest(unittest.TestCase):
|
||||
def test_accepts_multiple_group_ids(self):
|
||||
args = crawl.parse_args(['-100123', '-100456'])
|
||||
|
||||
self.assertEqual(args.group_ids, [-100123, -100456])
|
||||
|
||||
def test_allows_no_group_ids(self):
|
||||
args = crawl.parse_args([])
|
||||
|
||||
self.assertEqual(args.group_ids, [])
|
||||
|
||||
def test_rejects_non_integer_group_id(self):
|
||||
with (
|
||||
patch.object(crawl, 'load_app_config') as load_app_config,
|
||||
patch.object(crawl, 'TelegramClient') as telegram_client,
|
||||
self.assertRaises(SystemExit),
|
||||
):
|
||||
crawl.main(['not-a-group'])
|
||||
|
||||
load_app_config.assert_not_called()
|
||||
telegram_client.assert_not_called()
|
||||
|
||||
|
||||
class CrawlMainTest(unittest.TestCase):
|
||||
@patch.object(crawl, 'save_group_export')
|
||||
@patch.object(crawl, 'save_session')
|
||||
@patch.object(crawl, 'load_session', return_value='stored-session')
|
||||
@patch.object(crawl, 'load_app_config')
|
||||
@patch.object(crawl, 'TelegramClient')
|
||||
def test_crawls_configured_group_ids(
|
||||
self,
|
||||
telegram_client,
|
||||
load_app_config,
|
||||
load_session,
|
||||
save_session,
|
||||
save_group_export,
|
||||
):
|
||||
load_app_config.return_value = {
|
||||
'api_id': 123,
|
||||
'api_hash': 'hash',
|
||||
'phone': '+15551234567',
|
||||
'group_ids': [101, 202],
|
||||
}
|
||||
client = MagicMock()
|
||||
telegram_client.return_value = client
|
||||
client.is_user_authorized.return_value = True
|
||||
client.session.save.return_value = 'updated-session'
|
||||
client.get_entity.side_effect = [FakeChat(101), FakeChat(202)]
|
||||
client.get_participants.side_effect = [
|
||||
[FakeMember(2), FakeMember(1)],
|
||||
[FakeMember(4), FakeMember(3)],
|
||||
]
|
||||
|
||||
with (
|
||||
patch.object(crawl, 'Chat', FakeChat),
|
||||
patch.object(crawl, 'Channel', FakeChat),
|
||||
patch.object(crawl, 'StringSession', return_value='session'),
|
||||
patch.object(crawl, 'new_run_time', return_value='20260724120000'),
|
||||
):
|
||||
result = crawl.main([])
|
||||
|
||||
self.assertEqual(result, 0)
|
||||
self.assertEqual(client.get_entity.call_args_list, [call(101), call(202)])
|
||||
save_session.assert_called_once_with('updated-session')
|
||||
self.assertEqual(save_group_export.call_count, 2)
|
||||
first_members = save_group_export.call_args_list[0].args[2]
|
||||
self.assertEqual([member.id for member in first_members], [1, 2])
|
||||
|
||||
@patch.object(crawl, 'save_group_export')
|
||||
@patch.object(crawl, 'save_session')
|
||||
@patch.object(crawl, 'load_session', return_value='stored-session')
|
||||
@patch.object(crawl, 'load_app_config')
|
||||
@patch.object(crawl, 'TelegramClient')
|
||||
def test_supplied_ids_override_configured_ids(
|
||||
self,
|
||||
telegram_client,
|
||||
load_app_config,
|
||||
load_session,
|
||||
save_session,
|
||||
save_group_export,
|
||||
):
|
||||
load_app_config.return_value = {
|
||||
'api_id': 123,
|
||||
'api_hash': 'hash',
|
||||
'phone': '+15551234567',
|
||||
'group_ids': [999],
|
||||
}
|
||||
client = MagicMock()
|
||||
telegram_client.return_value = client
|
||||
client.is_user_authorized.return_value = True
|
||||
client.session.save.return_value = 'updated-session'
|
||||
client.get_entity.side_effect = [FakeChat(101), FakeChat(202)]
|
||||
client.get_participants.side_effect = [
|
||||
[FakeMember(2), FakeMember(1)],
|
||||
[FakeMember(4), FakeMember(3)],
|
||||
]
|
||||
|
||||
with (
|
||||
patch.object(crawl, 'Chat', FakeChat),
|
||||
patch.object(crawl, 'Channel', FakeChat),
|
||||
patch.object(crawl, 'StringSession', return_value='session'),
|
||||
patch.object(crawl, 'new_run_time', return_value='20260724120000'),
|
||||
):
|
||||
result = crawl.main(['101', '202'])
|
||||
|
||||
self.assertEqual(result, 0)
|
||||
self.assertEqual(client.get_entity.call_args_list, [call(101), call(202)])
|
||||
self.assertNotIn(call(999), client.get_entity.call_args_list)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -1,337 +0,0 @@
|
||||
import csv
|
||||
import io
|
||||
import types
|
||||
import unittest
|
||||
from contextlib import redirect_stderr, redirect_stdout
|
||||
from pathlib import Path
|
||||
from tempfile import TemporaryDirectory
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import export as export_command
|
||||
|
||||
|
||||
def export_record(group_id=-100123, run_time='20260724120000'):
|
||||
return {
|
||||
'group_id': group_id,
|
||||
'title': 'Test Group',
|
||||
'time': run_time,
|
||||
'members': [
|
||||
{
|
||||
'id': 1,
|
||||
'username': 'alice',
|
||||
'first_name': 'Alice',
|
||||
'last_name': 'Example',
|
||||
},
|
||||
{
|
||||
'id': 2,
|
||||
'username': None,
|
||||
'first_name': 'Bob',
|
||||
'last_name': None,
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
class ExportCommandTest(unittest.TestCase):
|
||||
def test_exports_exact_group_and_time(self):
|
||||
record = export_record()
|
||||
common = types.SimpleNamespace(
|
||||
get_group_export=MagicMock(return_value=record),
|
||||
latest_group_export_time=MagicMock(),
|
||||
)
|
||||
|
||||
with TemporaryDirectory() as temp_dir:
|
||||
output_dir = Path(temp_dir) / 'output'
|
||||
stdout = io.StringIO()
|
||||
with (
|
||||
patch.dict('sys.modules', {'common': common}),
|
||||
patch.object(export_command, 'OUTPUT_DIR', output_dir),
|
||||
redirect_stdout(stdout),
|
||||
):
|
||||
result = export_command.main(['-100123', '20260724120000'])
|
||||
|
||||
output_path = output_dir / '-100123-20260724120000.csv'
|
||||
self.assertEqual(result, 0)
|
||||
common.get_group_export.assert_called_once_with(
|
||||
-100123,
|
||||
'20260724120000',
|
||||
)
|
||||
common.latest_group_export_time.assert_not_called()
|
||||
self.assertTrue(output_path.is_file())
|
||||
self.assertIn(str(output_path), stdout.getvalue())
|
||||
|
||||
with output_path.open(encoding='utf-8', newline='') as csv_file:
|
||||
rows = list(csv.DictReader(csv_file))
|
||||
|
||||
self.assertEqual(
|
||||
csv.DictReader(io.StringIO(output_path.read_text())).fieldnames,
|
||||
export_command.CSV_FIELDS,
|
||||
)
|
||||
self.assertEqual(rows[0], {
|
||||
'id': '1',
|
||||
'username': 'alice',
|
||||
'first_name': 'Alice',
|
||||
'last_name': 'Example',
|
||||
})
|
||||
self.assertEqual(rows[1]['username'], '')
|
||||
self.assertEqual(rows[1]['last_name'], '')
|
||||
|
||||
def test_uses_latest_export_when_time_is_omitted(self):
|
||||
latest = export_record(run_time='20260724120000')
|
||||
common = types.SimpleNamespace(
|
||||
get_group_export=MagicMock(return_value=latest),
|
||||
latest_group_export_time=MagicMock(
|
||||
return_value='20260724120000',
|
||||
),
|
||||
)
|
||||
|
||||
with TemporaryDirectory() as temp_dir:
|
||||
output_dir = Path(temp_dir) / 'output'
|
||||
with (
|
||||
patch.dict('sys.modules', {'common': common}),
|
||||
patch.object(export_command, 'OUTPUT_DIR', output_dir),
|
||||
redirect_stdout(io.StringIO()),
|
||||
):
|
||||
result = export_command.main(['-100123'])
|
||||
|
||||
self.assertEqual(result, 0)
|
||||
common.latest_group_export_time.assert_called_once_with(-100123)
|
||||
common.get_group_export.assert_called_once_with(
|
||||
-100123,
|
||||
'20260724120000',
|
||||
)
|
||||
self.assertTrue(
|
||||
(output_dir / '-100123-20260724120000.csv').is_file(),
|
||||
)
|
||||
|
||||
def test_uses_first_configured_group_when_both_args_are_omitted(self):
|
||||
record = export_record(group_id=-100999)
|
||||
common = types.SimpleNamespace(
|
||||
get_group_export=MagicMock(return_value=record),
|
||||
latest_group_export_time=MagicMock(
|
||||
return_value='20260724120000',
|
||||
),
|
||||
)
|
||||
config = types.SimpleNamespace(
|
||||
load_app_config=MagicMock(return_value={'group_ids': [-100999]}),
|
||||
)
|
||||
|
||||
with TemporaryDirectory() as temp_dir:
|
||||
with (
|
||||
patch.dict('sys.modules', {'common': common, 'config': config}),
|
||||
patch.object(
|
||||
export_command,
|
||||
'OUTPUT_DIR',
|
||||
Path(temp_dir) / 'output',
|
||||
),
|
||||
redirect_stdout(io.StringIO()),
|
||||
):
|
||||
result = export_command.main([])
|
||||
|
||||
self.assertEqual(result, 0)
|
||||
config.load_app_config.assert_called_once_with()
|
||||
common.latest_group_export_time.assert_called_once_with(-100999)
|
||||
common.get_group_export.assert_called_once_with(
|
||||
-100999,
|
||||
'20260724120000',
|
||||
)
|
||||
|
||||
def test_default_group_config_failure_returns_clean_error(self):
|
||||
common = types.SimpleNamespace(
|
||||
get_group_export=MagicMock(),
|
||||
latest_group_export_time=MagicMock(),
|
||||
)
|
||||
config = types.SimpleNamespace(
|
||||
load_app_config=MagicMock(
|
||||
side_effect=RuntimeError('Redis unavailable'),
|
||||
),
|
||||
)
|
||||
stderr = io.StringIO()
|
||||
|
||||
with (
|
||||
patch.dict('sys.modules', {'common': common, 'config': config}),
|
||||
redirect_stderr(stderr),
|
||||
):
|
||||
result = export_command.main([])
|
||||
|
||||
self.assertEqual(result, 1)
|
||||
self.assertIn('could not read export configuration', stderr.getvalue())
|
||||
common.latest_group_export_time.assert_not_called()
|
||||
common.get_group_export.assert_not_called()
|
||||
|
||||
def test_common_import_failure_returns_clean_error(self):
|
||||
stderr = io.StringIO()
|
||||
real_import = __import__
|
||||
|
||||
def fail_common_import(name, *args, **kwargs):
|
||||
if name == 'common':
|
||||
raise ValueError('Redis URL is invalid')
|
||||
return real_import(name, *args, **kwargs)
|
||||
|
||||
with (
|
||||
patch('builtins.__import__', side_effect=fail_common_import),
|
||||
redirect_stderr(stderr),
|
||||
):
|
||||
result = export_command.main(['-100123', '20260724120000'])
|
||||
|
||||
self.assertEqual(result, 1)
|
||||
self.assertIn('could not read export configuration', stderr.getvalue())
|
||||
|
||||
def test_missing_exact_export_returns_error_without_creating_output(self):
|
||||
common = types.SimpleNamespace(
|
||||
get_group_export=MagicMock(return_value=None),
|
||||
latest_group_export_time=MagicMock(),
|
||||
)
|
||||
|
||||
with TemporaryDirectory() as temp_dir:
|
||||
output_dir = Path(temp_dir) / 'output'
|
||||
stderr = io.StringIO()
|
||||
with (
|
||||
patch.dict('sys.modules', {'common': common}),
|
||||
patch.object(export_command, 'OUTPUT_DIR', output_dir),
|
||||
redirect_stderr(stderr),
|
||||
):
|
||||
result = export_command.main(['-100123', '20260724120000'])
|
||||
|
||||
self.assertEqual(result, 1)
|
||||
self.assertFalse(output_dir.exists())
|
||||
self.assertIn('export not found', stderr.getvalue())
|
||||
|
||||
def test_missing_group_history_returns_error_without_creating_output(self):
|
||||
common = types.SimpleNamespace(
|
||||
get_group_export=MagicMock(),
|
||||
latest_group_export_time=MagicMock(return_value=None),
|
||||
)
|
||||
|
||||
with TemporaryDirectory() as temp_dir:
|
||||
output_dir = Path(temp_dir) / 'output'
|
||||
stderr = io.StringIO()
|
||||
with (
|
||||
patch.dict('sys.modules', {'common': common}),
|
||||
patch.object(export_command, 'OUTPUT_DIR', output_dir),
|
||||
redirect_stderr(stderr),
|
||||
):
|
||||
result = export_command.main(['-100123'])
|
||||
|
||||
self.assertEqual(result, 1)
|
||||
self.assertFalse(output_dir.exists())
|
||||
self.assertIn('no exports found', stderr.getvalue())
|
||||
|
||||
def test_rejects_invalid_time_before_loading_redis(self):
|
||||
for invalid_time in (
|
||||
'../unsafe',
|
||||
'20261301120000',
|
||||
'20260230010101',
|
||||
'20260724246000',
|
||||
):
|
||||
with self.subTest(invalid_time=invalid_time):
|
||||
with (
|
||||
patch.dict('sys.modules', {'common': None}),
|
||||
self.assertRaises(SystemExit),
|
||||
):
|
||||
export_command.main(['-100123', invalid_time])
|
||||
|
||||
def test_rejects_mismatched_record_without_creating_output(self):
|
||||
record = export_record(group_id='../escaped')
|
||||
common = types.SimpleNamespace(
|
||||
get_group_export=MagicMock(return_value=record),
|
||||
latest_group_export_time=MagicMock(),
|
||||
)
|
||||
|
||||
with TemporaryDirectory() as temp_dir:
|
||||
output_dir = Path(temp_dir) / 'output'
|
||||
stderr = io.StringIO()
|
||||
with (
|
||||
patch.dict('sys.modules', {'common': common}),
|
||||
patch.object(export_command, 'OUTPUT_DIR', output_dir),
|
||||
redirect_stderr(stderr),
|
||||
):
|
||||
result = export_command.main(['-100123', '20260724120000'])
|
||||
|
||||
self.assertEqual(result, 1)
|
||||
self.assertFalse(output_dir.exists())
|
||||
self.assertIn('group ID does not match', stderr.getvalue())
|
||||
|
||||
def test_rejects_impossible_time_from_redis_key(self):
|
||||
record = export_record(run_time='99999999999999')
|
||||
common = types.SimpleNamespace(
|
||||
get_group_export=MagicMock(return_value=record),
|
||||
latest_group_export_time=MagicMock(
|
||||
return_value='99999999999999',
|
||||
),
|
||||
)
|
||||
|
||||
with TemporaryDirectory() as temp_dir:
|
||||
output_dir = Path(temp_dir) / 'output'
|
||||
stderr = io.StringIO()
|
||||
with (
|
||||
patch.dict('sys.modules', {'common': common}),
|
||||
patch.object(export_command, 'OUTPUT_DIR', output_dir),
|
||||
redirect_stderr(stderr),
|
||||
):
|
||||
result = export_command.main(['-100123'])
|
||||
|
||||
self.assertEqual(result, 1)
|
||||
self.assertFalse(output_dir.exists())
|
||||
self.assertIn('invalid crawl time', stderr.getvalue())
|
||||
|
||||
def test_rejects_malformed_members_without_replacing_existing_csv(self):
|
||||
record = export_record()
|
||||
record['members'] = None
|
||||
common = types.SimpleNamespace(
|
||||
get_group_export=MagicMock(return_value=record),
|
||||
latest_group_export_time=MagicMock(),
|
||||
)
|
||||
|
||||
with TemporaryDirectory() as temp_dir:
|
||||
output_dir = Path(temp_dir) / 'output'
|
||||
output_dir.mkdir()
|
||||
output_path = output_dir / '-100123-20260724120000.csv'
|
||||
output_path.write_text('existing export', encoding='utf-8')
|
||||
with (
|
||||
patch.dict('sys.modules', {'common': common}),
|
||||
patch.object(export_command, 'OUTPUT_DIR', output_dir),
|
||||
redirect_stderr(io.StringIO()),
|
||||
):
|
||||
result = export_command.main(['-100123', '20260724120000'])
|
||||
|
||||
self.assertEqual(result, 1)
|
||||
self.assertEqual(
|
||||
output_path.read_text(encoding='utf-8'),
|
||||
'existing export',
|
||||
)
|
||||
|
||||
def test_escapes_spreadsheet_formulas_and_uses_private_permissions(self):
|
||||
record = export_record()
|
||||
record['members'][0]['first_name'] = ' =HYPERLINK("https://example.com")'
|
||||
record['members'][0]['last_name'] = '\n+SUM(1,1)'
|
||||
common = types.SimpleNamespace(
|
||||
get_group_export=MagicMock(return_value=record),
|
||||
latest_group_export_time=MagicMock(),
|
||||
)
|
||||
|
||||
with TemporaryDirectory() as temp_dir:
|
||||
output_dir = Path(temp_dir) / 'output'
|
||||
with (
|
||||
patch.dict('sys.modules', {'common': common}),
|
||||
patch.object(export_command, 'OUTPUT_DIR', output_dir),
|
||||
redirect_stdout(io.StringIO()),
|
||||
):
|
||||
result = export_command.main(['-100123', '20260724120000'])
|
||||
|
||||
output_path = output_dir / '-100123-20260724120000.csv'
|
||||
with output_path.open(encoding='utf-8', newline='') as csv_file:
|
||||
rows = list(csv.DictReader(csv_file))
|
||||
|
||||
self.assertEqual(result, 0)
|
||||
self.assertEqual(
|
||||
rows[0]['first_name'],
|
||||
'\' =HYPERLINK("https://example.com")',
|
||||
)
|
||||
self.assertEqual(rows[0]['last_name'], "'\n+SUM(1,1)")
|
||||
self.assertEqual(output_dir.stat().st_mode & 0o777, 0o700)
|
||||
self.assertEqual(output_path.stat().st_mode & 0o777, 0o600)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
Reference in new issue
Block a user