docs(plans): add image importer enhancements plan

Six phases spanning resize, on-canvas overlay preview, transforms, more
dithering algorithms, color correction, and skip-white/paint-transparent
toggles. Based on WPlace-AutoBOT's image-processor.js feature set, scoped
to what fits our 32-color canvas.
This commit is contained in:
tiennm99 committed 2026-04-17 11:43:04 +07:00
1 parent 4a149c38eb
commit 9254fec8c9
7 files changed
+400

No files matched your search

@@ -0,0 +1,74 @@
# Phase 01 — Resize Controls
## Overview
- **Priority:** P0 (must-have, user requested)
- **Status:** Done
- Let the user change the output image size *before* it is palette-converted and uploaded, with a choice of resampling method so pixel art and photos both look good.
## Key Insights
- Canvas is 2048x2048 and images often don't fit. Forcing the user to pre-resize externally is friction.
- Resampling choice matters on our tiny palette: nearest preserves sharp pixel art; bilinear/box avoids aliasing on photos.
- Resize must happen *before* palette quantization — quantizing then scaling produces garbage.
- Aspect-lock prevents accidentally squishing logos; free mode supports deliberate stretch.
- HTMLCanvas `drawImage(scaled)` already provides nearest + bilinear cheaply. Box/median/dominant are pixel-art-friendly and need manual loops (see WPlace's `resampleBox`, `resampleMedian`, `resampleDominant`).
## Requirements
- Width + height number inputs, with a lock-aspect-ratio toggle.
- "Fit to canvas" helper that caps to `CANVAS_WIDTH`/`CANVAS_HEIGHT` minus current origin.
- Resampling dropdown: `nearest`, `bilinear`, `box`, `median`, `dominant` (ship at minimum `nearest` + `bilinear` in this phase; `box`/`median`/`dominant` optional).
- Reactive: changing any resize input re-runs pipeline and updates preview.
- Must keep preview snappy on 512x512 inputs (< 100ms).
## Architecture
New module `src/lib/image-resize.js` exports:
- `resizeRgba(rgba, srcW, srcH, dstW, dstH, method) → Uint8ClampedArray` — pure function operating on RGBA buffers so it's framework-free and CLI-usable.
`ImageImporter.svelte` adds resize state (`resizeW`, `resizeH`, `lockAspect`, `resampleMethod`) and inserts resize as the first pipeline step before `rgbaToPalette`.
```
srcRgba (srcW×srcH)
→ resizeRgba(rgba, srcW, srcH, resizeW, resizeH, method)
→ rgbaToPalette(resized, resizeW, resizeH, { dither })
```
## Related Code Files
**Modify:**
- `src/client/components/ImageImporter.svelte` — add UI + pipeline wiring
- `scripts/image-to-colors.js` — add `--width`, `--height`, `--method` flags (DRY with lib)
**Create:**
- `src/lib/image-resize.js` — resize function(s)
- `test/lib/image-resize.test.js` — unit tests
## Implementation Steps
1. Write `src/lib/image-resize.js` with `resampleNearest`, `resampleBilinear` (both via OffscreenCanvas `imageSmoothingEnabled`). Add `resampleBox` and `resampleMedian` only if time permits (optional stretch).
2. Export `resizeRgba(rgba, srcW, srcH, dstW, dstH, method = 'nearest')` — dispatches to the right resampler and returns a fresh `Uint8ClampedArray`.
3. Add unit tests: identity-resize (same dims) returns equivalent data; down/up scale by integer factors produce expected shape; unknown method falls back to nearest.
4. In `ImageImporter.svelte`:
- Add state `resizeW`, `resizeH`, `lockAspect`, `resampleMethod` (default source dims, lock on, nearest).
- When file loads, init `resizeW/resizeH` to source dims.
- Reactive `$derived` or `$effect` computes `workingRgba` = `resizeRgba(srcRgba, srcW, srcH, resizeW, resizeH, method)`.
- Feed `workingRgba` into `rgbaToPalette`. Preview canvas uses `resizeW/resizeH`.
- Aspect-lock updates the other dim when one changes.
- "Fit to canvas" button clamps to `CANVAS_WIDTH - originX` / `CANVAS_HEIGHT - originY` preserving aspect.
5. Update validation: overflow check uses `resizeW/resizeH`, not source dims.
6. CLI: add `--width`, `--height`, `--method` flags to `scripts/image-to-colors.js`, routing through the same `resizeRgba`. Keep defaults = source dims so behavior unchanged.
## Todo
- [x] `src/lib/image-resize.js` with `resizeRgba` + nearest/bilinear/box resamplers
- [x] Unit tests `test/lib/image-resize.test.js` (8 tests)
- [x] Wire into `ImageImporter.svelte` (state, pipeline, UI controls, aspect lock, "Fit to canvas", "1:1")
- [x] CLI flags in `scripts/image-to-colors.js` (`--width`, `--height`, `--method`)
- [x] `npm run build` + `npm test` green (84/84 tests)
## Success Criteria
- Upload a 512x512 photo, resize to 128x128 in-app, see palette preview update within one paint, upload fills a 128x128 area on canvas correctly.
- `node scripts/image-to-colors.js foo.png --width 128 --height 128 --method bilinear` produces same output pixel count (`128*128 = 16384`).
- All 76 existing tests still pass; new tests green.
## Risks
- Forgetting to rebase the pipeline on `workingRgba` everywhere → stale preview vs upload. Mitigation: single `$effect` owns the pipeline end-to-end; `buildPixels` reads the same `paletteIndices`.
- Very large upscales (e.g. 2000x2000) blow compute on preview re-render. Mitigation: cap resize inputs at `CANVAS_WIDTH`/`CANVAS_HEIGHT`.
## Next Steps
- Phase 2 builds on the final `resizeW/resizeH` to overlay the preview on the main canvas at `(originX, originY)`.
@@ -0,0 +1,61 @@
# Phase 02 — Overlay Preview on Canvas
## Overview
- **Priority:** P1 (high UX value)
- **Status:** Todo
- Show a semi-transparent ghost of the palette-converted image on the main canvas at `(originX, originY)`, so the user can zoom/pan and confirm alignment *before* spending credits.
## Key Insights
- Today the user sees the preview in the panel only — they have to guess-and-upload, which wastes credits on misalignment.
- Ghost overlay must live at the canvas layer, not in the importer, because the user pans/zooms the canvas freely.
- Overlay must not interfere with manual drawing mode.
- Overlay opacity toggle (e.g. 50%) helps differentiate it from committed pixels.
## Requirements
- `CanvasRenderer` accepts an optional overlay: `{ x, y, width, height, indices }` (same `Int16Array` the importer already builds).
- Rendered as a second layer after committed + pending, at configurable alpha.
- Toggle on/off from the importer panel.
- Updates on importer changes (resize/dither/etc.) with no jank.
## Architecture
Extend `CanvasRenderer` with:
- A dedicated OffscreenCanvas for the overlay, same dims as the overlay's bounding box.
- `setOverlay({ x, y, width, height, indices, alpha } | null)` method.
- Render pipeline: commit/pending → offscreen → main canvas → `ctx.globalAlpha = alpha; ctx.drawImage(overlay, x, y);`.
`ImageImporter.svelte` holds a toggle `showOverlay` (default true); a reactive `$effect` calls `canvasRenderer.setOverlay(...)` whenever `paletteIndices` / `originX` / `originY` / `showOverlay` changes, and clears on close.
## Related Code Files
**Modify:**
- `src/client/components/CanvasRenderer.svelte` — overlay layer + method
- `src/client/components/ImageImporter.svelte` — show/hide toggle, wiring
- `src/client/App.svelte` — pass `canvasRenderer` ref to importer (already has bind, may need forwarding)
## Implementation Steps
1. In `CanvasRenderer`, allocate `overlayCanvas: OffscreenCanvas | null = null`. On `setOverlay(o)`:
- If `o == null`, drop the overlay and re-render.
- Else, build an `ImageData` from `paletteToRgba(o.indices, o.width, o.height)`, paint it on a fresh OffscreenCanvas, store `overlayState = { x, y, canvas, alpha }`.
2. Extend `render()` to draw the overlay after `offscreen`, respecting `globalAlpha`.
3. Expose `setOverlay` via `export function`.
4. In `ImageImporter`, add `showOverlay` checkbox (default true) and an opacity slider (default 0.5).
5. Wire a `$effect` that calls `setOverlay(showOverlay && paletteIndices ? { x: originX, y: originY, width: resizeW, height: resizeH, indices: paletteIndices, alpha } : null)` on relevant changes.
6. Clear overlay on panel close and when upload finishes successfully (optional; keep if user wants to re-align).
## Todo
- [ ] `CanvasRenderer.setOverlay` + render integration
- [ ] Importer toggle + alpha slider
- [ ] Reactive wiring (importer ↔ renderer)
- [ ] Verify overlay clears on close / unmount
## Success Criteria
- Toggle overlay on → preview appears on the canvas at the chosen position at 50% alpha.
- Move origin X/Y → overlay follows in real time.
- Disable overlay → canvas returns to normal appearance.
- No regression in existing draw/paint/undo/redo flows.
## Risks
- Performance on large overlays (up to canvas size): limit overlay dims to `resizeW*resizeH <= some cap` (e.g. 1M pixels) or always use OffscreenCanvas (GPU-accelerated).
- Race with WebSocket updates changing `committedColors` — overlay is drawn on top so it's fine.
## Next Steps
- Phase 3 (transforms) will rotate/flip the indices; overlay must re-render on transform toggle.
@@ -0,0 +1,50 @@
# Phase 03 — Transforms (Flip / Rotate)
## Overview
- **Priority:** P2 (medium)
- **Status:** Todo
- Let the user flip horizontally, flip vertically, and rotate in 90° increments without re-exporting the source.
## Requirements
- Buttons: Flip H, Flip V, Rotate CW 90°, Rotate CCW 90°, Reset.
- Cumulative state (internal): `{ flipH: bool, flipV: bool, rotation: 0|90|180|270 }`.
- Rotations by 90 swap width/height; resize controls must reflect post-transform dims.
## Architecture
New module `src/lib/image-transform.js`:
- `transformRgba(rgba, w, h, { flipH, flipV, rotation }) → { rgba, width, height }`
- Pure function, reusable by CLI.
Applied in pipeline **before** resize so resize targets the post-transform orientation:
```
srcRgba → transformRgba → resizeRgba → rgbaToPalette
```
## Related Code Files
- `src/lib/image-transform.js` (new)
- `src/client/components/ImageImporter.svelte` (toolbar row + state)
- `scripts/image-to-colors.js` (`--rotate 90 --flip-h --flip-v` flags)
- `test/lib/image-transform.test.js` (new)
## Implementation Steps
1. Implement pure transform on flat RGBA: flips are in-row or inter-row swaps; rotations reindex `(x,y) → (y, w-1-x)` etc.
2. Unit-test for all combinations — a known 2x3 grid rotated/flipped and compared pixel-exact.
3. Wire transform state + buttons in importer; pipeline insertion before resize.
4. CLI flags + doc update.
## Todo
- [ ] `image-transform.js` + tests
- [ ] Importer buttons + state
- [ ] CLI flags
- [ ] Verify preview, overlay (Phase 2), and upload agree
## Success Criteria
- Flip/rotate buttons produce visually correct preview.
- `npm test` green.
- CLI output byte-identical between rotating in-app and rotating via CLI.
## Risks
- Low. Pure-function transforms with tests cover correctness.
## Next Steps
- None blocking; proceed to Phase 4 or 5 independently.
@@ -0,0 +1,55 @@
# Phase 04 — More Dithering Algorithms
## Overview
- **Priority:** P2 (medium)
- **Status:** Todo
- Add Atkinson, Jarvis, Stucki, Burkes, Sierra variants (error diffusion) and Bayer 2x2/4x4/8x8 ordered dithering so users can pick the visual style that fits their source.
## Key Insights
- Today we only ship Floyd-Steinberg. Atkinson produces softer output (good for photos). Bayer (ordered) creates the classic 8-bit texture (good for retro art).
- All error-diffusion kernels share the same inner loop — differ only in the kernel weights. Factor that out.
- Ordered dithering is a different algorithm: add a matrix-indexed bias before quantizing; no error propagation.
## Requirements
- Dropdown replaces the current dither checkbox: `none`, `floyd`, `atkinson`, `jarvis`, `stucki`, `burkes`, `sierra`, `sierra-lite`, `bayer-2`, `bayer-4`, `bayer-8`.
- Optional `strength` slider 0–1 for error-diffusion blends (default 1).
## Architecture
Refactor `src/lib/image-to-palette.js`:
- Extract a `runErrorDiffusion(rgba, w, h, alphaThreshold, kernel)` with the current Floyd-Steinberg loop generalized to take a kernel `[{ dx, dy, w }, …]`.
- Add `runOrderedDither(rgba, w, h, alphaThreshold, matrix)` using Bayer matrices.
- `rgbaToPalette(rgba, w, h, { alphaThreshold, method })` dispatches on `method`.
- Deprecate `dither: bool` option but keep it mapped to `method: 'floyd'` for back-compat with the CLI until the next breaking release.
## Related Code Files
- `src/lib/image-to-palette.js` (refactor)
- `src/client/components/ImageImporter.svelte` (dropdown + strength slider)
- `scripts/image-to-colors.js` (`--method` flag replaces `--dither`)
- `test/lib/image-to-palette.test.js` (extend)
## Implementation Steps
1. Extract shared kernel runner. Move FS kernel into a table.
2. Add Atkinson, Jarvis, Stucki, Burkes, Sierra, SierraLite kernels (copy weights from WPlace).
3. Add Bayer matrices (2x2, 4x4, 8x8) and the ordered-dither function.
4. Map UI dropdown → `method` option.
5. Extend unit tests: each method runs without throwing, returns correct length, transparent pixels preserved.
6. Visual regression: snapshot-test a fixed gradient across methods.
## Todo
- [ ] Refactor FS into generalized error-diffusion runner
- [ ] Add Atkinson/Jarvis/Stucki/Burkes/Sierra/SierraLite kernels
- [ ] Add Bayer 2/4/8 ordered dithering
- [ ] UI dropdown + strength slider
- [ ] CLI `--method`, keep `--dither` as alias
- [ ] Tests
## Success Criteria
- Each method produces distinct, visibly reasonable output on a test gradient.
- Existing `--dither` CLI flag still works (mapped to `method=floyd`).
- Build + tests green.
## Risks
- Kernel-weight typos produce wrong but still-plausible output. Mitigation: unit-test kernel sums equal 1.0.
## Next Steps
- None blocking.
@@ -0,0 +1,60 @@
# Phase 05 — Color Correction Sliders
## Overview
- **Priority:** P2 (medium)
- **Status:** Todo
- Sliders to adjust brightness, contrast, saturation, and gamma so photos map better onto our 32-color palette.
## Key Insights
- The 32-color palette is narrow. A slightly dark photo can palette-quantize entirely to black+dark-grey. Bumping brightness/contrast recovers detail.
- Saturation matters because our palette has vivid primaries — desaturating a photo first avoids neon-looking output.
- Gamma interacts well with dithering: mid-gamma lets dithering distribute error over mid-tones.
## Requirements
- Sliders:
- Brightness: -100 to +100 (default 0)
- Contrast: -100 to +100 (default 0)
- Saturation: -100 to +100 (default 0)
- Gamma: 0.1 to 3.0 (default 1.0)
- Reset button.
- Reactive — moving a slider re-quantizes and re-renders preview.
## Architecture
New module `src/lib/image-color-correction.js`:
- `applyColorCorrection(rgba, w, h, { brightness, contrast, saturation, gamma }) → Uint8ClampedArray`
- Pure, framework-free.
Inserted in pipeline after transform, before resize (applies at source resolution, better fidelity):
```
srcRgba → transform → colorCorrect → resize → palette
```
(Order note: applying after resize is cheaper but loses precision on gamma — keep at source res unless preview lag becomes a problem. If so, switch to post-resize.)
## Related Code Files
- `src/lib/image-color-correction.js` (new)
- `src/client/components/ImageImporter.svelte` (slider group)
- `scripts/image-to-colors.js` (flags: `--brightness`, `--contrast`, `--saturation`, `--gamma`)
- `test/lib/image-color-correction.test.js` (new)
## Implementation Steps
1. Implement brightness/contrast/saturation/gamma per-pixel. Reference: WPlace `applyColorCorrection`. Convert RGB→HSV for saturation, adjust S, HSV→RGB.
2. Unit tests: brightness +100 saturates to 255; gamma 1 is identity; etc.
3. Add slider group in importer with live reactive updates (debounce ~50ms if jank).
4. CLI flags.
## Todo
- [ ] `image-color-correction.js` + tests
- [ ] Sliders + reset button
- [ ] CLI flags
- [ ] Verify no jank on 512×512 with live slider
## Success Criteria
- Default sliders → output identical to previous pipeline (regression guard).
- Brightness/contrast/saturation/gamma each visibly change output in expected direction.
- Build + tests green.
## Risks
- Perf: re-running full pipeline per slider tick. Mitigation: debounce, or operate on post-resize buffer once resize is stable.
## Next Steps
- None blocking.
@@ -0,0 +1,57 @@
# Phase 06 — Skip-White / Paint-Transparent Toggles
## Overview
- **Priority:** P3 (low, easy win)
- **Status:** Todo
- Two cheap toggles: treat near-white pixels as transparent (useful for logos on white backgrounds), and optionally paint pixels with alpha < threshold as white instead of skipping.
## Requirements
- Checkbox "Skip white pixels" + threshold (0–255, default 230).
- Checkbox "Paint transparent pixels as white" (default off).
- Reactive pipeline updates.
## Architecture
Extend `rgbaToPalette` options:
```js
rgbaToPalette(rgba, w, h, {
alphaThreshold,
method,
skipWhite: false,
whiteThreshold: 230,
paintTransparent: false,
});
```
Logic:
- `paintTransparent=true`: alpha < threshold → treat as `(255,255,255,255)` instead of `-1`.
- `skipWhite=true`: if `r,g,b >= whiteThreshold` → output `-1` (skip).
- These are mutually compatible and commute (transparent→white happens first; white-skip sees the filled-in whites).
## Related Code Files
- `src/lib/image-to-palette.js` (extend options)
- `src/client/components/ImageImporter.svelte` (checkboxes + threshold slider)
- `scripts/image-to-colors.js` (`--skip-white`, `--white-threshold`, `--paint-transparent`)
- `test/lib/image-to-palette.test.js` (extend)
## Implementation Steps
1. Extend the `rgbaToPalette` options object; defaults preserve current behavior.
2. Update both `quantizeNearest` and error-diffusion paths to check the new conditions per pixel.
3. UI checkboxes with a small threshold input next to skip-white.
4. CLI flags.
5. Tests: near-white pixel → -1 iff skipWhite on; low-alpha pixel → -1 iff paintTransparent off.
## Todo
- [ ] Extend `rgbaToPalette` options
- [ ] UI toggles + threshold
- [ ] CLI flags
- [ ] Tests
## Success Criteria
- With defaults, output matches pre-change output.
- With skipWhite on, a white-background logo uploads only the logo (no background pixels queued).
- Build + tests green.
## Risks
- Order of operations between skipWhite and paintTransparent — tests pin this down.
## Next Steps
- After Phase 6 ships, revisit parked items: multi-algorithm color distance (Lab/Oklab), Kuwahara, template save/load, repair mode.
@@ -0,0 +1,43 @@
# Image Importer Enhancements
Reference: [WPlace-AutoBOT image-processor.js](https://github.com/Wplace-AutoBot/WPlace-AutoBOT/blob/main/Extension/scripts/image-processor.js).
## Goal
Grow the in-app Image Import panel into a full pre-placement pipeline: resize, transforms, tune, quantize with multiple dither algorithms, overlay-preview on canvas, then upload.
## Principles
- YAGNI/KISS — only add the features with clear UX value on our 2048x2048, 32-color canvas.
- DRY — all pixel transforms live in `src/lib/image-*.js` and are reused by the CLI (`scripts/image-to-colors.js`).
- Keep reactivity driven by `$effect` over a cached source RGBA buffer; each toggle re-runs the pipeline without re-decoding the file.
- Never block manual drawing; uploader continues sharing the server-side rate limit.
## Phases
| # | Phase | Status | Value |
|---|---|---|---|
| 1 | [Resize controls](phase-01-resize-controls.md) | Done | Must-have — user explicitly asked |
| 2 | [Overlay preview on canvas](phase-02-overlay-preview.md) | Todo | High — visualize alignment before spending credits |
| 3 | [Transforms (flip / rotate)](phase-03-transforms.md) | Todo | Medium — quick fixes without re-editing source |
| 4 | [More dithering algorithms](phase-04-dithering-algorithms.md) | Todo | Medium — different looks per source |
| 5 | [Color correction sliders](phase-05-color-correction.md) | Todo | Medium — photos benefit most |
| 6 | [Skip-white + paint-transparent toggles](phase-06-skip-white.md) | Todo | Low — easy win, often useful for logos |
Later / parked: multi-algorithm color distance (Lab/Oklab), Kuwahara smoothing, template save/load, repair mode. Revisit after Phase 6.
## Dependencies
- Phase 1 (resize) blocks Phase 2 (overlay needs final dimensions) and Phase 3 (transforms sit before or after resize, but the pipeline shape is set by Phase 1).
- Phases 4, 5, 6 are independent of each other.
## Shared pipeline shape (locked after Phase 1)
```
File → decode → srcRgba
↓ (reactive on: resize/transforms/color-correction/dither/skip-white)
pipeline()
↓
Int16Array paletteIndices
↓
preview (palette → RGBA)
+ buildPixels(originX, originY, skipMatching) → upload
```
All steps operate on RGBA buffers so they compose. `rgbaToPalette` stays the final step.