mirror of
https://github.com/tiennm99/telegram-exporter.git
synced 2026-10-11 03:13:49 +00:00
tiennm99
aea29826c2
Verification matched on name and non-zero size, so an upload that died partway was counted archived permanently. Comparing against the size Telegram reports found six such objects in the live archive, one of them 221 MiB standing in for a 2006 MiB video. They are folded into the outstanding set; rclone overwrites a size mismatch, so another pass repairs them. A zero Report claimed the archive was complete — nothing expected, nothing missing — which is the value both commands hold before their Telegram callback populates it, so any early return printed COMPLETE and exited 0 on an untouched chat. A Report now knows whether it ran. A name that can never be written kept the run outstanding forever while the download set deliberately excluded it, so a driver looping on "incomplete" walked the whole history and re-indexed the whole remote every pass for work that could not be done. Such a run now reports STALLED and exits 4. A destination that stopped accepting uploads was reported and then discarded, exiting 1. The same driver would retry against a full or unreachable remote indefinitely, downloading gigabytes each pass to upload none. It exits 3. Free space is re-checked during the run, not only before it. An archive this size runs for hours, and the remote can fill in the middle; discovering it through five failed multi-gigabyte uploads wastes the download for all of them. Also: parseSize silently wrapped to a negative or zero on a large input, which reads downstream as "no cap"; list printed attacker-chosen filenames raw, so a tab shifted the columns and an escape sequence reached the terminal; a missing backend blamed credentials rather than the build; humanBytes indexed past its unit table above 1 PiB; a second Init reported success against a config that never loaded; and sync did not surface the basename collisions verify warned about, though sync is the command that acts on the verdict. The env-override test could not observe what it claimed: rclone reads RCLONE_* at package init, so t.Setenv came too late and the assertion held with the guard removed. It runs in a subprocess now, as does the new one covering the index against inherited filters.
67 lines
1.9 KiB
Go
67 lines
1.9 KiB
Go
package pipeline
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"sync"
|
|
"time"
|
|
)
|
|
|
|
// spaceCheckInterval is how often the destination's free space is re-read
|
|
// mid-run. About is a network round trip, so it is not worth doing per file.
|
|
const spaceCheckInterval = time.Minute
|
|
|
|
// spaceGuard watches the destination's free space during a run.
|
|
//
|
|
// Checking only before starting is not enough on a long archive: an 18k-message
|
|
// chat runs for hours, and a remote that was fine at the start can fill in the
|
|
// middle — from this run's own uploads, or from anything else using the account.
|
|
// Without this the run discovers it by failing several multi-gigabyte uploads in
|
|
// a row, which costs the download bandwidth for all of them.
|
|
//
|
|
// A backend that cannot report a quota is treated as unlimited, matching the
|
|
// pre-flight check and the shell pipeline before it.
|
|
type spaceGuard struct {
|
|
free func(context.Context) (int64, bool)
|
|
minFree int64
|
|
|
|
mu sync.Mutex
|
|
lastCheck time.Time
|
|
failed error
|
|
}
|
|
|
|
func newSpaceGuard(free func(context.Context) (int64, bool), minFree int64) *spaceGuard {
|
|
if free == nil || minFree <= 0 {
|
|
return nil
|
|
}
|
|
return &spaceGuard{free: free, minFree: minFree, lastCheck: time.Now()}
|
|
}
|
|
|
|
// check reports an error once the destination has dropped below the floor.
|
|
//
|
|
// The verdict is sticky: once the remote is known to be full, every later call
|
|
// says so without another round trip, because the run is ending either way.
|
|
func (g *spaceGuard) check(ctx context.Context) error {
|
|
if g == nil {
|
|
return nil
|
|
}
|
|
g.mu.Lock()
|
|
defer g.mu.Unlock()
|
|
|
|
if g.failed != nil {
|
|
return g.failed
|
|
}
|
|
if time.Since(g.lastCheck) < spaceCheckInterval {
|
|
return nil
|
|
}
|
|
g.lastCheck = time.Now()
|
|
|
|
free, ok := g.free(ctx)
|
|
if !ok || free >= g.minFree {
|
|
return nil
|
|
}
|
|
g.failed = fmt.Errorf("%w: only %.1f GiB free, below the %.1f GiB floor",
|
|
ErrDestinationFailing, float64(free)/(1<<30), float64(g.minFree)/(1<<30))
|
|
return g.failed
|
|
}
|