Register GifAsset extension for .gif loading, create AnimatedGifSprite
wrapper with lazy load/unload matching ImageSprite pattern, and detect
.gif assets in SceneManager to use animated rendering. GIFs stored as
type 'image' in scene format — no schema changes needed.
New ExportDialog component with scope (selection/all), filename,
format (PNG/JPEG/WebP), quality slider, scale multiplier, and
background color picker. Wired into Toolbar and Editor.
1. URL imports now participate in upload manager pipeline:
addUrlJob() creates a job row, uploadComplete/setFailed called
on success/failure — consistent with local file uploads
2. activeCount only counts uploading/processing jobs, not failed.
Failed jobs no longer inflate the "Uploads (n)" header count.
3. uploadManager.clear() called on boardId change so stale jobs
from a previous board don't linger in the next board's panel.
- UploadManager store: tracks jobs through uploading → processing → done/failed
- UploadPanel component: floating bottom-left widget showing upload progress,
file names, sizes, status with auto-dismiss for completed items
- axios onUploadProgress wired for real-time upload percentage
- Video jobs transition to "processing" after upload, then "done" when
media:job:update socket event arrives (reuses existing pipeline)
- Failed uploads show error message from server response
- Clear button to dismiss finished/failed items
Pass SelectionManager to setupDragDrop and setupPaste. After each
upload completes, the newly created scene items are selected:
- Single file/URL: selectOnly
- Multi-file drop/paste: select all new items
Thread-centric model: one pin = one thread with flat replies.
Server-authoritative real-time sync (all clients receive canonical
events). Soft Review Mode for pin visibility without locking editing.
Denormalized thread summaries for efficient panel queries.
Public boards: read-only feedback, auth required to participate.
Phased: comments first, then canvas pins, then voting.
- Background media worker: polls media_jobs table, runs ffprobe+ffmpeg
with concurrency limit, generates video posters, emits socket events
- Non-blocking video upload: stores file + enqueues job, returns immediately
- Poster hydration on board load: GET /boards/:id injects poster/dimensions
from DB into canvas_state video objects
- SpatialGrid: fixed-cell (512px) spatial hash for O(nearby) culling instead
of O(all) item scanning, eliminates setTimeout violations
- Canvas-based video rendering: draws video frames to offscreen canvas then
uploads to GPU, completely eliminates GL_INVALID_OPERATION errors from
PixiJS VideoSource auto-update mechanism
- Server poster upgrade path: culling ticker and applyProcessedMedia() both
upgrade client-captured posters to server posters when available
- Pause restores server poster (paused video behaves like an image)
- Selection drag-end persistence: onObjectDragEnd broadcasts + saves + undo
- Live media:job:update socket handler patches scene data + VideoSprite
dimensions without broadcast fanout
Add _serverPosterLoading flag so repeated 200ms culling ticks don't
call textures.load() multiple times before the first resolves. Also
release the texture ref if the sprite was destroyed or started playing
during the async load.
1. Server posters no longer auto-load in VideoSprite constructor.
loadServerPoster() is now public and called by the culling system
only when within MAX_POSTER_VIDEOS budget. Videos outside the
budget stay as placeholders regardless of server poster availability.
2. video-utils cleanup() now uses statSync to distinguish files from
directories, unlinks files first, then rmdirs. Previously tmpDir
was passed through path.dirname() which resolved to /tmp instead
of the created temp directory.
Upload pipeline now extracts poster frame (JPEG) and metadata
(width, height, duration) from videos at upload time using ffmpeg.
Server:
- Add ffmpeg to Docker image (Alpine)
- video-utils.js: probeVideo() and extractPoster() using ffprobe/ffmpeg
- Upload response includes poster_asset_key and duration
Frontend:
- VideoObject gains poster and duration fields in scene format
- VideoSprite accepts posterAssetKey + TextureManager, loads poster
as a regular image texture on construction (no <video> needed)
- Server poster loaded/released via TextureManager ref counting
- addVideoFromUpload passes poster and duration through to scene data
- image-drop.ts forwards poster_asset_key from upload response
Result: videos with server posters render as images by default.
Zero <video> elements needed for thumbnails. Only explicit play
creates a media element.
preload='metadata' doesn't decode video frames, so readyState stays
at 1 (HAVE_METADATA) and loadeddata never fires. capturePoster() now
detects this state and forces a seek to 0.1s via _trySeekCapture(),
which triggers frame decode and captures on the 'seeked' event.
Video memory:
- Lazy <video> creation: constructor makes placeholder only, initVideo()
creates element with preload='metadata' when near viewport
- Tiered budgets: 1 playing / 6 initialized / 4 poster textures max
- Aggressive teardown: offscreen videos lose <video> element, poster,
and all textures — zero memory for offscreen videos
- Poster textures dropped while playing (don't keep both resident)
- Dimension caching in scene data (nativeW/nativeH) avoids re-init
Server:
- HTTP Range support (206 Partial Content) for video seeking
- Proper end clamping and invalid range rejection (416)
Sync:
- Remove redundant broadcastSceneDebounced() from broadcastTransform()
- Remove duplicate broadcastElements() from drag-end handler
VideoControls:
- Event-driven (timeupdate/play/pause/seeked) instead of rAF polling
- Tracks actual HTMLVideoElement reference, rebinds on init/teardown
- Seek uses ref-based commit on pointerUp, not conditional onChange
- Add onDragOver/onDrop preventDefault on container div (React level)
to always block browser default file-open behavior
- Add DOM fallback in useCanvasSetup if ref is somehow null
- setupDragDrop's native listeners handle the actual upload logic
The drag/drop setup was searching for a <canvas> element to find its
parent, which was fragile and could fail (returning null). When it
failed, document-level dragover/drop prevention was never registered,
so the browser's default behavior (open file in new tab) kicked in.
Fix: pass a direct ref to the outer canvas container div from Editor
to useCanvasSetup. No more DOM querying for the drop target.
When duplicated images share the same assetKey, Assets.unload()
was destroying the shared texture source — crashing all sprites
using it (null addressModeU/alphaMode in WebGL renderer).
TextureManager now tracks refCount per asset. Only calls
Assets.unload() when the last reference is released. ImageSprite
calls release() instead of unload() on texture teardown.
LRU eviction was destroying texture sources while sprites still
referenced them, causing PixiJS v8 'alphaMode of null' crash.
The viewport culling system (MAX_LOADED_TEXTURES=60) already controls
what's loaded/unloaded, so TextureManager doesn't need its own eviction.
Simplified to a pure load/unload cache with no budget tracking.
PixiJS v8 crashes with 'alphaMode of null' even when Sprite.visible
is false — it still traverses the display object. Fix: create the
Sprite lazily in loadTexture() only after a real texture is ready.
On unload, remove and destroy the Sprite entirely.
When zoomed out, all items fall within the viewport so the culling
system tried to load everything at once. Now:
- Max 60 textures loaded at any time (sorted by distance to center)
- Max 5 new texture loads per tick (prevents spike)
- Items over budget or far away are unloaded immediately
PixiJS v8 crashes with 'Cannot read alphaMode of null' when rendering
a Sprite with Texture.EMPTY. Fix: set sprite.visible = false until a
real texture is loaded, re-hide on unload.
Replace broadcastSceneNow() with incremental element:update for all
property-modifying operations (align, arrange, flip, resize, rotate,
normalize, frame color). Full scene sync now only fires via debounced
fallback (500ms) for structural changes (add/remove/paste/group).
Changes:
- onCanvasChange() accepts optional changedIds for incremental sync
- SelectionToolbar, context menu, shortcuts all pass item IDs
- TransformBox.onDragEnd passes item IDs instead of full scene
- ShortcutContext, MenuContext, hook interfaces updated
This dramatically reduces sync traffic during normal editing — only
the changed elements are sent instead of the entire board state.
Images no longer load textures eagerly on creation. Instead, the
culling ticker checks viewport proximity every 200ms:
- Load margin: 1x screen size beyond viewport (preload before visible)
- Unload margin: 2x screen size (hysteresis prevents thrash during pan)
ImageSprite changes:
- Constructor no longer calls loadTexture() — deferred to culling system
- New unloadTexture() restores placeholder and frees GPU memory
- Exposed texture getter for clipboard resolution detection
- loaded field now public for culling system to check state
This should allow hundreds of images without GPU OOM, since only
nearby textures consume GPU memory at any time.
- Skip thumbnail generation when scene has >50 items (extract.canvas
on full viewport with many textures causes GPU OOM and context loss)
- Add WebGL context lost/restored handlers — auto-reload scene on recovery
- Reduce texture memory budget from 512MB to 256MB for GPU headroom
- Make PixiJS canvas background transparent so SVG dot grid shows through
- Move dot grid SVG before canvas in DOM with zIndex 0 (behind images)
- Set wrapper div background to #1e1e1e
- Remove 600px cap on pasted images — preserve native dimensions
Remove server-side Sharp resize and texture dimension cap — PixiJS
handles large textures natively. Keep the proper Assets.unload() fix
to prevent "TextureSource destroyed instead of unloaded" warnings.
- Distribute H: normalizes height + arranges as row (uniform horizontal strip)
- Distribute V: normalizes width + arranges as column (uniform vertical strip)
- Also reduced min selection from 3 to 2 for both operations
When dropping multiple images, arranges them in a square-ish grid
(cols = ceil(sqrt(count))) with 20px gaps, tracking per-column widths
and per-row heights for proper alignment.
- Batch broadcastTransform: sends all selected items in one emit instead
of throttle-dropping all but the first (multi-select drag now syncs all)
- TransformBox onDragEnd: persist+sync final state after resize/rotate
(was only broadcasting ephemeral transforms, never the final position)
- Layer lock toggle: add missing onCanvasChange() call for sync+save
- Receiver handles both batched and legacy single-item transform formats
onCanvasChange now calls broadcastSceneNow() so rearrangement,
alignment, and other bulk operations sync to other tabs in real-time
instead of waiting for the next periodic sync or page reload.
Computes resolution multiplier from native texture size vs display size,
so a 2000px image shown at 400px on canvas copies close to 2000px.
Capped at 4x to avoid excessive memory usage.
Floating HTML overlay with play/pause, seekbar, time display, and
mute toggle that appears when a single video element is selected.
Positioned at the bottom of the video using viewport.toScreen()
coordinate conversion, polled at 100ms for position and 250ms for
playback progress.
Introduces TextEditor utility that overlays a styled <textarea> matching
the PixiJS Text object's font, size, and color. Hides the PixiJS text
while editing, commits on blur/Enter, cancels on Escape, and updates
dimensions from measured bounds on save.
- TransformBox: use item.data (world-space) instead of getBounds() (screen-space)
for correct handle positioning regardless of viewport pan/zoom
- TransformBox: scale drag sensitivity relative to object size, not fixed 500px
- TransformBox: compensate drag deltas for viewport zoom level
- TransformBox: fix corner/edge handles to keep opposite edge fixed during resize
- SelectionManager: use world-space hit testing instead of screen-space getBounds()
- SelectionManager: fix rubber band selection to use world-space item bounds
- TextureManager: detect legacy asset keys (with file extension) vs new LOD keys
and load original file directly for pre-migration images (GPU handles scaling)
dashed border and "Inbox" label. Auto-synced media slides in with
spring animations and random offset/rotation for a natural look.
MattermostImport: modal dialog for pulling images from a MM thread
URL and managing linked channels for auto-sync. Uses the existing
dark modal styling pattern from ShortcutsHelp.
Editor wiring: toolbar button for MM import, InboxZone added to
viewport on init, socket listener for board:media-arrived events
that routes assets into the InboxZone with a toast notification.
Runs a culling check every 200ms alongside the spring ticker.
Off-screen image sprites are hidden (visible=false), on-screen
ones get updateLOD() called with the current zoom level. Video
sprites get onVisibilityChange() to auto-play/pause based on
viewport visibility. Uses a 200px margin to avoid popping.
DropShadowFilter applied to all ImageSprites for photos-on-a-desk feel.
SelectionManager now handles object dragging with shadow lift/drop and
spring-animated scale (1.0 <-> 1.03). Replaced incompatible @pixi/filter-drop-shadow
v5 with pixi-filters v6 for PixiJS v8 compatibility.
- Add mm_file_id column to images table for dedup tracking
- mm-pull fetches posts from MM API, downloads media files, uploads with LOD to MinIO
- Register bridge routes in server.js under /api/boards
Children converge ~25px toward combined center on group (gentle spring),
then reparent into a Container. Ungroup reparents back to viewport and
spreads children ~25px outward with matching animation.
- Rewrite Editor.tsx: swap FabricCanvas for PixiCanvas, wire
SelectionManager, SceneManager, and all Pixi-based APIs
- Port tools.ts from Fabric Canvas API to PixiJS Viewport/Scene
- Update LayerPanel type icons for PixiJS scene types (text, video)
- Remove FabricCanvas.tsx (dead code after swap)
- Uninstall fabric npm package
- All operations, shortcuts, sync, history, drag/drop, paste now
flow through the PixiJS pipeline end-to-end
When a board is loaded via GET /api/boards/:id, if the canvas_state
lacks a v2 marker, it is converted in-place and persisted back to DB
so subsequent loads skip conversion.
Replace all Fabric.js Canvas references with SceneManager, SelectionManager,
UndoManager, and Viewport. Remove withBreak/breakSelection helper (no longer
needed without ActiveSelection). Implement z-swap layer ordering for ] and [
keys, clone-based paste/duplicate via SceneManager._createItem, and direct
history.undo()/redo() calls instead of undoRef indirection.
Rewrite sync module to work with SceneManager and v2 scene format
instead of Fabric.js. Key changes:
- setupSync takes SceneManager instead of Canvas
- broadcastScene uses sceneManager.serialize() for v2 SceneData
- broadcastTransform reads from item.data (x,y,sx,sy,angle)
- onSceneReceived uses diff-based loadScene (no flicker, no full reload)
- onTransformReceived updates item.data + displayObject directly
- Remove _suppress/resumeBroadcasts (diff-based loading is safe)
- Remove userInteracting/pendingScene deferral (not needed with diffs)
- Remove all Fabric.js imports and types
Replace Fabric.js Canvas dependency with SceneManager. UndoManager now
serializes via sceneManager.serialize() and restores via loadScene()
with diff-based reconciliation (no flicker, no CORS workarounds).
Replace all FabricObject usage with SceneItem, accessing item.data.x/y/w/h/sx/sy
instead of obj.left/top/width*scaleX. Sync displayObject position/scale after
mutations. Remove breakSelection (no ActiveSelection needed), setCoords calls,
and all fabric imports. Toggle grayscale uses PixiJS ColorMatrixFilter.desaturate().
arrangeByZOrder now uses item.data.z directly instead of canvas object index.
Replace Fabric Canvas/FabricImage/Rect with PixiJS Graphics and
pixi-viewport's toWorld() for coordinate mapping. Upload responses
now dispatch to SceneManager.addImageFromUpload() for images or
create VideoSprite directly for video media types based on the
backend's asset_key and media_type fields.
Implements inline video playback as a PixiJS Sprite subclass with:
- MAX_CONCURRENT_VIDEOS (3) with oldest-eviction policy
- Visibility-based auto play/pause for viewport culling
- Play icon overlay, tap-to-toggle-mute interaction
- Proper resource cleanup on destroy (video element + texture)
Introduces ImageSprite (extends Sprite) that manages per-image LOD
texture swapping via TextureManager. Shows a placeholder rect until
the first texture loads, then upgrades/downgrades tiers based on zoom.
Implements Task 3.1: manages the selected item set with hit testing
in reverse z-order, rubber-band rectangle selection on empty-space drag,
shift-click toggle, and viewport drag-pause during rubber banding.
Integrates with TransformBox to show handles on selection change.
Implements Task 3.2: an interactive transform overlay that draws
8 resize handles (corners + edge midpoints) and a rotation handle
around the combined bounding rect of selected scene items. Each handle
supports pointer-drag to apply scale and rotation transforms.
Implements the main React component wrapping PixiJS v8 Application and
pixi-viewport with: async init, viewport persistence per board,
space-to-pan, spring animation ticker, v1/v2 scene loading, ResizeObserver
resizing, and imperative handle (fitAll, zoom control, scene/viewport access).
Scene serialization layer for the PixiJS migration: defines SceneData v2
types (Image/Video/Text/Group objects) and convertFabricToV2() which
transforms existing Fabric canvas JSON to the new format. Handles asset
key extraction from /api/images/ URLs, group flattening with coordinate
transforms, filter extraction, and edge cases for missing fields.
GPU texture cache keyed by asset:tier that loads textures on demand
via PixiJS Assets.load(), estimates RGBA memory usage, and evicts
least-recently-used entries when the budget is exceeded.
Damped harmonic oscillator with configurable stiffness, damping, mass,
and precision. Includes four presets (drop, gentle, snappy, bounce) and
a SpringManager to batch-tick and auto-remove settled springs.
Queries DB for images where asset_key IS NULL, downloads each from
MinIO, generates LOD tiers (thumb/medium/full), uploads them, and
updates the DB record. Skips LOD for videos, SVGs, and GIFs. Logs
per-image progress and continues on individual failures.
- Upload route now generates 3-tier LOD (thumb/medium/full) for images via
lod-generator service, storing each tier at boards/{boardId}/{imageId}/
- Video uploads (mp4, webm, quicktime) supported as single-file storage
- New DB columns: asset_key (prefix path) and media_type ('image'/'video')
- Backward compatible: minio_path and public_url still populated (point to full tier)
- SVG and GIF skip LOD processing (single file stored)
- Added putBuffer() and MIME_TO_EXT export to minio.js
Sharp-based service that produces thumb (256px), medium (1024px),
and full (pass-through) tiers from an uploaded image buffer.
Skips upscaling when the source is smaller than a tier's width.
- Auto-generate board thumbnail from canvas content on save (webp, 400px)
- Collection cards show stacked/fanned board thumbnails with random offsets
- Three-dot menu on all cards with Rename, Share, Delete actions
- Inline rename for collections (header + card) and boards
- Collection cards show public/private status and member count
- Share button on collection cards for quick access
- Add updateCollection/updateBoard API functions
- Add thumbnail + object_count columns to boards table with migration
- Fix zoom drift: use element-relative coords (offsetX/Y) not getScenePoint
- Fix thumbnail generation breaking viewport: use finally block for restore
- Fix thumbnail bounds: use scene-space obj coords not screen-space getBoundingRect
- Fix object counter: live count from canvas instead of stale DB images table
- Fix three-dot menu clipping by removing overflow:hidden from card containers
- Status bar shows "objects" instead of "images", updates on add/delete
Viewport transform (zoom level + pan offset) is saved to localStorage
keyed by board ID. Restored automatically when reopening the same board.
Debounced 300ms save on zoom/pan to avoid excessive writes.
- Copy/paste/duplicate now uses obj.clone() instead of broken fromObject() serialization
- Ctrl+C stores actual object refs, Ctrl+V clones them — works for images, groups, paths, text
- System clipboard copy (Ctrl+C / Ctrl+Shift+C) properly re-selects after screenshot
- Layer panel: double-click to rename, collapsible groups with expand/collapse arrow
- Auto-indexed names: Image 1, Image 2, Drawing 1, Drawing 2, etc.
- Group children visible in layer panel when expanded
- Copy/paste works on groups and mixed multi-selections
- Bot auto-replies in DMs without @root mention (dm_history.py for rolling context)
- chat_with_history() in llm_client for multi-turn DM conversations
- Fixed MM v11 healthcheck (curl removed from image, now uses mmctl)
- RefBoard PUBLIC_URL set to refboard.unleashtheavatargame.com
- Added RefBoard to cloudflared dependencies
- Updated ROOT-AI-STACK.md: MM 11.4, 19 services, RefBoard, DM support
- RefBoard PRD (docs/plans/refboard-prd.md)
- Editor.tsx: improved clone logic using Fabric.js enqueueObject
Full-featured PureRef-style collaborative canvas for game dev teams:
- Layer panel with visibility, lock, drag reorder, group/ungroup (Ctrl+G/Shift+G)
- Arrangement tools (grid, row, column) via right-click context menu
- Copy to system clipboard (Ctrl+C writes PNG for external paste in Paint etc.)
- Number shortcuts (1-5) for tool selection with visible shortcut badges
- Premium dark UI across all pages (Login, Collections, Boards, Editor)
- Socket.IO rooms for cursors, transforms, and presence notifications
- MinIO image storage with backend proxy, drag/drop and paste upload
Hiren Kangad