mirror of
https://github.com/tiennm99/rplace.git
synced 2026-10-13 10:16:02 +00:00
tiennm99
9f50237a3c
Drop /admin/migrate-from-upstash from public API docs, replace deprecated UPSTASH_* env vars with ALLOWED_ORIGINS/ENVIRONMENT, update Worker conventions to reflect DO bindings (env.CANVAS_ROOM) instead of Redis client patterns, and rewrite system-architecture security/data-flow sections to cover cookie+IP identity, transactionSync atomicity, WS Origin allowlist, per-identity WS cap, and broadcast sequence numbers.
167 lines
6.8 KiB
Markdown
167 lines
6.8 KiB
Markdown
# System Architecture
|
||
|
||
## Overview
|
||
|
||
rplace is a collaborative pixel canvas deployed as a single Cloudflare Worker.
|
||
The frontend (Svelte SPA) is served as static assets, and a single
|
||
**Durable Object** (`CanvasRoom`, `idFromName('main')`) owns canvas state,
|
||
rate-limit cooldowns, and the WebSocket broadcast hub. The Worker is a
|
||
thin validation/routing proxy.
|
||
|
||
## Component Map
|
||
|
||
```
|
||
Browser (Svelte SPA + WebSocket)
|
||
| GET /api/canvas → 16 MB binary, edge-cached 10s
|
||
| POST /api/place → batch pixel placement (validated at edge)
|
||
| WS /api/ws → real-time pixel deltas
|
||
v
|
||
Cloudflare Worker (Hono — thin proxy)
|
||
└─▶ CanvasRoom Durable Object (single instance, 'main')
|
||
├── canvas_chunks SQLite BLOB rows (256 × 64 KB = 16 MB)
|
||
├── cooldowns SQLite TTL rows (1s rate-limit)
|
||
└── WebSocket hub Hibernation API broadcasts pixel deltas
|
||
```
|
||
|
||
## Data Flow
|
||
|
||
### Pixel Placement
|
||
|
||
```
|
||
1. User draws on canvas → optimistic render into a pending buffer.
|
||
2. User hits Submit → POST /api/place { pixels: [{x, y, color}, ...] }.
|
||
3. Worker validates input (bounds, types, batch ≤ 2048, Content-Length).
|
||
4. Worker resolves identity (cookie or IP fallback) and forwards to DO /place.
|
||
5. DO atomically:
|
||
a. cooldowns.tryAcquire(userId, 1s) — UPDATE expired or INSERT new row.
|
||
On conflict (active claim), responds 429.
|
||
b. state.storage.transactionSync(() => chunk_storage.writePixels(...))
|
||
— group pixels by chunk_id, read each touched chunk's BLOB, modify
|
||
in memory, INSERT OR REPLACE. Transaction makes a multi-chunk batch
|
||
all-or-nothing.
|
||
c. On write failure, cooldown is refunded so transient errors don't
|
||
soft-DOS the user.
|
||
d. Broadcast `{type:'pixels', seq, pixels}` to all hibernating WebSockets.
|
||
6. Response: { ok: true } (or { error, retryAfter } on rate limit).
|
||
```
|
||
|
||
The DO is single-threaded; the cooldown + write + broadcast sequence runs
|
||
without preemption. The transactionSync wrapper provides atomicity across
|
||
multiple BLOB chunks.
|
||
|
||
### Canvas Loading
|
||
|
||
```
|
||
1. Client fetches GET /api/canvas (10s edge-cache; CDN serves 99% of hits).
|
||
2. Worker forwards to DO /canvas on miss.
|
||
3. DO chunk_storage.readAllChunks: SELECT all canvas_chunks rows, copy each
|
||
BLOB into a single 16 MB Uint8Array offset by chunk_id × CHUNK_BYTES.
|
||
4. Client receives raw bytes (CF auto-gzip), maps each byte → COLORS_RGBA.
|
||
5. Renders via OffscreenCanvas + ImageData.
|
||
```
|
||
|
||
### Real-time Updates
|
||
|
||
```
|
||
1. Client connects WS /api/ws.
|
||
2. Worker forwards upgrade to DO /ws (URL rewritten to /ws so the DO
|
||
pathname dispatch matches).
|
||
3. DO state.acceptWebSocket(server) — Hibernation API; idle sockets
|
||
survive DO eviction at zero CPU cost.
|
||
4. On placement → DO iterates state.getWebSockets() and sends JSON.
|
||
5. Client merges received pixels into local ImageData; re-renders dirty
|
||
region.
|
||
6. On disconnect → exponential-backoff reconnect (1 s → 30 s).
|
||
```
|
||
|
||
## Storage
|
||
|
||
### `canvas_chunks` (canvas pixels)
|
||
|
||
| Column | Type | Notes |
|
||
|---|---|---|
|
||
| `chunk_id` | INTEGER PRIMARY KEY | 0 .. CHUNK_COUNT − 1 |
|
||
| `bytes` | BLOB NOT NULL | Exactly CHUNK_BYTES bytes (last chunk may be short) |
|
||
|
||
- Linear byte layout: `offset = y * CANVAS_WIDTH + x`
|
||
- `chunk_id = floor(offset / CHUNK_BYTES)`
|
||
- `CHUNK_BYTES = 65536`, `CHUNK_COUNT = ceil(TOTAL_PIXELS / CHUNK_BYTES)`
|
||
- **Lazy initialization:** missing rows read as zero-filled buffers. Resizing
|
||
the canvas is just a constants change — new chunks materialize on first
|
||
read. No migration script needed.
|
||
- **Hard caps:** CF DO BLOB row size 2 MB → 64 KB chunks have 32× headroom.
|
||
|
||
### `cooldowns` (rate-limit windows)
|
||
|
||
| Column | Type | Notes |
|
||
|---|---|---|
|
||
| `user_id` | TEXT PRIMARY KEY | `cookie:<uuid>` or `ip:<sha256-prefix>` |
|
||
| `expires_at` | INTEGER NOT NULL | ms epoch; row becomes stale past this point |
|
||
|
||
- `idx_cooldowns_expires` keeps lazy GC sweeps cheap.
|
||
- GC runs at 1% sample rate inside `tryAcquire`, wrapped in `try/catch` —
|
||
best-effort; never blocks the rate-limit decision.
|
||
|
||
## Rate Limiting
|
||
|
||
Single-row per user, 1 s window. Race-safe inside the DO via:
|
||
|
||
```
|
||
UPDATE cooldowns SET expires_at = ? WHERE user_id = ? AND expires_at <= ?
|
||
→ if rowsWritten > 0: claim acquired (existing row was expired).
|
||
→ else: INSERT INTO cooldowns ...
|
||
if INSERT throws on PK conflict → claim denied (active row exists).
|
||
```
|
||
|
||
Hardness comes from the DO single-threaded model; no SQL-level locking
|
||
needed.
|
||
|
||
## Free-tier Footprint (CF, 2026)
|
||
|
||
| Resource | Quota | rplace usage at hobby scale | Headroom |
|
||
|---|---|---|---|
|
||
| Workers requests | 100K/day | ~100/day (50 users) | 1000× |
|
||
| DO storage / object | 10 GB | 16 MB | 600× |
|
||
| DO storage / account | 5 GB | 16 MB | 300× |
|
||
| BLOB row size | 2 MB | 64 KB chunks | 32× |
|
||
| Per-DO request rate | 1,000 / s soft cap | <1 / s | 1000× |
|
||
| WS connections / DO | tens of thousands | ~50 | huge |
|
||
|
||
`/api/canvas` carries `Cache-Control: public, s-maxage=10, stale-while-revalidate=30`.
|
||
Verify edge HIT in production via `cf-cache-status: HIT`; if absent, wrap
|
||
the worker handler with the Cache API to enforce caching.
|
||
|
||
## Security
|
||
|
||
- **Rate limiting**: race-safe at the actor (single-threaded DO); writes
|
||
wrapped in `state.storage.transactionSync` so a multi-chunk batch is
|
||
all-or-nothing.
|
||
- **Identity**: opaque `rplace_id` cookie (HttpOnly, Secure, SameSite=Lax,
|
||
1y) issued on first `/api/canvas`. Falls back to a SHA-256 prefix of
|
||
`cf-connecting-ip` when no cookie is present. In production a request
|
||
with neither returns 500 `no_identity`.
|
||
- **Cooldown refund**: failed writes refund the cooldown row so a
|
||
transient storage error doesn't soft-DOS the user for 1 s.
|
||
- **Input validation**: strict bounds + type checks at the worker edge,
|
||
re-validated at the DO trust boundary. `Content-Length` required on
|
||
POST `/api/place` (411 if missing/zero, 413 if above cap).
|
||
- **Body cap**: ~128 KB per `/api/place` (2048 pixels × ~64 B JSON each).
|
||
- **WS Origin allowlist**: `ALLOWED_ORIGINS` env var (comma-separated)
|
||
rejects upgrades from disallowed origins; empty = allow all (dev).
|
||
- **Per-identity WS cap**: 5 concurrent sockets per identity prevents
|
||
broadcast amplification; further upgrades return 429.
|
||
- **DO isolation**: `/canvas`, `/place`, `/ws` are intra-DO paths only —
|
||
not internet-reachable except via the worker.
|
||
|
||
## Operational Notes
|
||
|
||
- **Single-region.** DO is anchored to one Cloudflare colo; latency for
|
||
far-away users mirrors the previous Upstash Redis topology — no
|
||
regression.
|
||
- **DO eviction mid-place** is safe: the SQLite write commits before
|
||
broadcast. If the DO evicts before broadcast fires, client WS receives
|
||
no message but auto-reconnects and re-fetches `/api/canvas`. No data
|
||
loss; minor latency tail.
|
||
- **Storage billing.** Per-account 5 GB free cap on Free plan as of
|
||
Jan 7 2026 — current 16 MB is far under. Monitor on resize.
|