Artist Program · Technical Reference

IKANDY Scene Contract

Status: v2 — drafted Phase 2.3 (2026-05-21), revised 2026-05-26 to fold in the WebGPU compute category as a first-class, fully-built category. The v1 draft listed webgpu as "reserved but not implemented"; that is now stale — the ComputeRuntime (dispatch orchestrator, audio UBO, storage-buffer pool, pipeline manager) is live and 8 compute scenes ship on it. Documents the architectural contract the curated pipeline scenes implement collectively.

Relationship to scene-schema.md: that doc is the data spec — what fields a scene may declare. This doc is the runtime contract — what a scene must do behind those fields, how it relates to the pipeline runtime, who owns which GPU resources, and what each category's render pattern looks like in code. Read both for a complete picture; a scene that satisfies the schema but violates the contract still won't ship.


Section 0 — Foundational principle

IKANDY is a music visualizer. Every scene must be demonstrably audio-reactive.

Time-only animations, static visuals, and ambient backdrops are out of scope by definition — not "discouraged" or "low priority," but not part of the product category. A beautiful scene that doesn't visibly respond to audio belongs in ShaderToy, not IKANDY.

This principle drives downstream decisions:

The remaining sections describe the engineering contract that lets a scene satisfy this rule without breaking the runtime.


Section 1 — What "compliant" means

A scene is contract-compliant when it:

  1. Satisfies Section 0 — visibly responds to audio under the harness reactivity check
  2. Declares the required identity fields (id, name, credit, category, schema)
  3. Implements an init(runtime) → SceneInstance matching its category's pattern
  4. Returns a SceneInstance with the required lifecycle methods (update, resize, dispose)
  5. Renders into runtime.hdrTarget (or the appropriate bridge for its category)
  6. Respects the declared color-space contract (selfTonemapped flag truthfully describes its output)
  7. Cleans up the GPU resources it created on dispose()
  8. Does not retain references to the runtime, GL context, or HDR target after dispose() returns
  9. Does not touch any DOM element outside its own offscreen canvas (canvas2d) or render target (three)

The pipeline runtime trusts items 2–9. Item 1 is enforced by the harness, not the runtime. Violating 2–9 silently corrupts state across scene transitions or leaks GPU resources; violating 1 means the scene isn't an IKANDY scene at all and shouldn't have been merged.


Section 2 — Lifecycle (all categories)

Compute (category: 'webgpu') scenes use a frame(ctx) hook, not update(audio, dt) — see §5. The flow below is the pipeline categories (fragment / canvas2d / three).

register(scene)            ← module load
  ↓
setActive(id)              ← user picks scene (or harness drives)
  → instance = scene.init(runtimeApi)   // sync OR async; runtime awaits
  ↓
loop:
  instance.update(audio, dt)            // ~once per frame
  ↓
resize(w, h)               ← viewport change
  → instance.resize(w, h)
  ↓
clearActive() / setActive(otherId)
  → instance.dispose()

Once dispose() returns, the runtime treats the scene as fully gone. It may then activate the next scene and reuse the GL context for new programs/textures/FBOs without any state cleanup. If your scene leaks (uncleared gl.useProgram, dangling textures bound to active units, modified blend/depth state without restore), the next scene inherits the corruption.

The Three.js runtime is the one exception — it calls _renderer.resetState() after every update() to fence Three.js's internal state changes from leaking. Fragment and canvas2d scenes don't get this safety net; they must leave GL state clean themselves. In practice the adapters handle this — see "Per-category patterns" below.


Section 3 — The runtime object passed to init()

Source: src/render/pipeline-runtime.js near L879.

runtime = {
  gl,                  // WebGL2RenderingContext, shared with every scene
  canvas,              // the pipeline canvas (#pipeline-canvas); category:'three'
                       // scenes wrap this with THREE.WebGLRenderer
  hdrTarget,           // { fbo, tex, width, height } — the HDR FBO scenes render INTO
  width, height,       // backing-pixel dimensions (CSS px × dpr)
  dpr,                 // devicePixelRatio used to size the FBO
  audioGetter: () => _audio,   // CURRENT audio frame; canvas2d scenes typically
                               // don't call this — audio arrives via update(audio, dt)
  helpers,             // { VS_FULLSCREEN, compileShader, makeProgram, drawFullscreen,
                       //   bindTargetForDraw } — minimal GL utilities scenes can use
  bindHDR(),           // shortcut for bindTargetForDraw(gl, hdrTarget)
  controls,            // schema:2 — current values keyed by ControlSpec.target
                       // (populated from defaults + localStorage before init runs)
};

gl is shared. Every scene uses the same WebGL2 context. There is no second context for a scene to opt into. (Pre-Phase-1.4 each scene owned its own context; the Chromium ~16-context cap was the architectural ceiling that motivated the pipeline.)

hdrTarget is a RGBA16F half-float FBO in linear color space. Scenes write linear-light values; the pipeline tonemaps + applies sRGB at the output pass (passes 3 + 9 of the chain). Exception: selfTonemapped: true scenes write sRGB-encoded [0,1] directly into the same FBO and the pipeline's tonemap pass is bypassed for them. See "Color space" below.

helpers.makeProgram(gl, fragmentSource [, vertexSource]) compiles a fragment shader against the shared fullscreen vertex shader (VS_FULLSCREEN). Scenes that need a custom vertex shader pass it as the third arg; the GLSL adapter handles this transparently for the 14 fragment scenes.


Section 4 — The SceneInstance returned from init()

Every category returns the same shape:

interface SceneInstance {
  update(audio: AudioFrame, dt: number): void;
  resize(w: number, h: number): void;
  dispose(): void;
}

audio shape is defined in scene-schema.md § AudioFrame. dt is wall-clock seconds since the previous tick; scenes that want time scaled by the user's speed slider should use audio.scaledDt instead (already = dt × audio.speed).

update() must:

resize(w, h) may be a no-op if the scene has nothing to resize. Most fragment scenes return resize() {}; canvas2d scenes typically call bridge.resize(w, h); three scenes update their render target.

dispose() must free everything init() allocated. This is the only hard rule that the runtime cannot enforce for you. Audit a scene's init() for each gl.create*()/new THREE.*()/document.createElement('canvas') and ensure each has a matching cleanup in dispose().


Section 5 — Per-category patterns

The curated scenes fall into four categories — fragment, canvas2d, three, and webgpu (WebGPU compute). Each has a different runtime relationship; new scenes must implement the same pattern as the existing scenes in their category.

One structural distinction up front: the first three categories render into runtime.hdrTarget (directly for fragment, via a bridge for canvas2d/three) on the shared WebGL2 context. The fourth — webgpu — runs on a separate GPUDevice and renders to its own #wgpu-canvas; the pipeline pulls that canvas in as a source (setWgpuSourceCanvas()) rather than receiving FBO writes. Read its subsection for why the contract there is shaped differently.

category: 'fragment' — GLSL (13 scenes — 12 fullscreen-quad + ferrofluid, a 3D mesh; see §11)

Scenes: aurora, eeu, fic, hol, hypnotise, ikandy-eye, jaf, lmi, prism, singularity, sw, voyager, ferrofluid, plus the four canvas2d adjacent scenes that are GLSL-rendered (see canvas2d section).

Pattern: never write a fragment scene by hand. Use the GLSL adapter.

import { createGlslScene } from './glsl-adapter.js';

export const myScene = createGlslScene({
  id:       'my-scene',
  name:     'My Scene',
  credit:   'Author Name',
  category: 'fragment',
  schema:   2,
  hints:    { selfTonemapped: true, audioReactive: true, /* … */ },
  audio:    { bass: true, mid: true, treble: true, intensity: true, time: true },
  controls: [ /* ControlSpec[] */ ],
  fragmentSource: `#version 300 es
    precision highp float;
    uniform float u_time;
    uniform float u_bass;
    // …
    out vec4 outColor;
    void main() { outColor = vec4(/* … */); }
  `,
});

The adapter (src/render/scenes/glsl-adapter.js) handles:

Standard uniforms the adapter binds (declared = bound; absent = skipped):

Uniform Source Range
iTime / u_time scaledTime accumulator (+= dt × audio.speed) seconds
iResolution / u_res vec2(hdrTarget.width, hdrTarget.height) backing px
iMouse / u_mouse mouse position × dpr, Y-flipped to GL bottom-origin px
u_bass / u_mid / u_treble audio.bass/mid/treble [0, 1]
u_intensity / u_sensitivity audio.intensity [0, ~2]
u_volume volume slider value [0, 2]
u_bassSmooth / u_midSmooth / u_trebSmooth smoothed bands [0, 1]
u_beat / u_transient / u_onset / u_energy / u_tempo audio events (Phase 2.7) [0, 1]

Two naming conventions are accepted in parallel: iTime/iResolution/iMouse (ShaderToy / mrange CC0 ports) and u_time/u_res/u_mouse (IKANDY originals). Declare either; both routes write the same value.

Don't write a fragment scene that bypasses the adapter. A scene that compiles its own shader, binds uniforms directly, and writes to the FBO outside the adapter loses harness time-pinning, the smoothed-bands state, persistent control plumbing, and texture-load fallback. Future contract changes (new standard uniforms, new validation, new compile-time checks) will land in the adapter and miss bypassed scenes.

category: 'canvas2d' — Canvas 2D drawn into HDR FBO (4 scenes)

Scenes: fire, pawrticles, ripple, waveray.

Pattern: use the canvas2d bridge.

import { createCanvas2dBridge } from '../runtimes/canvas2d-runtime.js';

export const myScene = {
  id: 'my-scene', name: 'My Scene', credit: 'Author',
  category: 'canvas2d', schema: 2,
  hints: { selfTonemapped: true, audioReactive: true },
  audio: { /* … */ },
  controls: [ /* … */ ],

  init(runtime) {
    const { gl, helpers, hdrTarget } = runtime;
    const bridge = createCanvas2dBridge(gl, helpers, hdrTarget.width, hdrTarget.height);
    const ctx = bridge.ctx;
    return {
      update(audio, dt) {
        // Draw with ctx (CanvasRenderingContext2D)
        ctx.fillStyle = 'rgba(0,0,0,0.5)';
        ctx.fillRect(0, 0, bridge.canvas.width, bridge.canvas.height);
        // … scene-specific drawing …
        bridge.blitToHdr(runtime.hdrTarget);   // upload + sample into HDR FBO
      },
      resize(w, h) { bridge.resize(w, h); },
      dispose()    { bridge.dispose(); },
    };
  },
};

The bridge (src/render/runtimes/canvas2d-runtime.js) handles:

Per-scene canvases are intentional (vs. one shared canvas). Two canvas2d scenes might want different blend modes / composite operations / global alpha; isolated canvases prevent cross-contamination.

Canvas2d scenes are always selfTonemapped: true. Canvas 2D outputs sRGB pixels by design; the bridge uploads them as-is and the pipeline's tonemap pass is bypassed. Writing linear values via a canvas2d scene is not supported.

audio.scaledDt vs dt: canvas2d scenes accumulate time manually for animation (no equivalent of the adapter's _scaledTime). Use audio.scaledDt ?? dt for animation that should respond to the speed slider; use raw dt for physics that should stay wall-clock.

category: 'three' — Three.js scene + camera (1 scene)

Scenes: trails.

Pattern: use the three runtime.

import * as THREE from 'three';
import {
  ensureRenderer, ensureSceneTarget,
  renderToSceneTarget, blitToHdr,
} from '../runtimes/three-runtime.js';

export const myScene = {
  id: 'my-scene', name: 'My Scene', credit: 'Author',
  category: 'three', schema: 2,
  hints: { selfTonemapped: false, /* see Color Space */ },
  audio: { /* … */ },
  controls: [ /* … */ ],

  init(runtime) {
    const { gl, canvas, helpers, hdrTarget } = runtime;
    const renderer = ensureRenderer(gl, canvas, helpers);
    const target = ensureSceneTarget(hdrTarget.width, hdrTarget.height);

    const scene  = new THREE.Scene();
    const camera = new THREE.PerspectiveCamera(50, hdrTarget.width / hdrTarget.height, 0.1, 1000);

    // sRGB-authored colors MUST be converted to linear at construction —
    // see "Color input contract" below for the full rationale.
    const mat = new THREE.MeshBasicMaterial({
      color: new THREE.Color('#ff5900').convertSRGBToLinear(),
    });
    // … build scene graph using `mat`, other geometries, etc. …

    return {
      update(audio, dt) {
        // animate scene graph using audio + dt
        renderToSceneTarget(scene, camera);       // renders into the shared HalfFloat target
        blitToHdr(runtime.hdrTarget, helpers);    // copies that target into the pipeline FBO
      },
      resize(w, h) {
        ensureSceneTarget(w, h);
        camera.aspect = w / h;
        camera.updateProjectionMatrix();
      },
      dispose() {
        // Free YOUR geometries / materials / textures here. Do NOT call
        // renderer.dispose() — the renderer is shared across three scenes
        // and lives for the lifetime of the pipeline runtime.
        scene.traverse(obj => {
          if (obj.geometry) obj.geometry.dispose();
          if (obj.material) {
            const mats = Array.isArray(obj.material) ? obj.material : [obj.material];
            for (const m of mats) m.dispose();
          }
        });
      },
    };
  },
};

The three runtime (src/render/runtimes/three-runtime.js) handles:

The renderer is shared. Do not call renderer.dispose() from a scene. That would tear down the pipeline's GL context.

Color space for three scenes is special. With renderer toneMapping: NoToneMapping + outputColorSpace: LinearSRGBColorSpace, the scene writes linear values to the HalfFloat target. The blit copies them into the HDR FBO unchanged, where the pipeline's ACES pass tonemaps + sRGBs them. So three scenes are selfTonemapped: false. Trails sets this; future three scenes should as well. (Setting selfTonemapped: true on a three scene means the pipeline will skip its tonemap pass, leaving linear values in an sRGB output — wrong, and visually obvious as washed-out shadows.)

Color input contract — read this before writing a three scene

DO NOT pass sRGB-encoded hex colors directly into THREE.Color and use them as material inputs. Under our renderer's outputColorSpace: LinearSRGBColorSpace configuration, Three.js does NOT auto-convert THREE.Color values from sRGB to linear during shader uniform binding (it would under the default SRGBColorSpace). An unconverted sRGB hex triplet flows into the shader as if it were already linear — quietly wrong, and downstream tonemap + bloom math compounds the error.

This is the specific bug that almost shipped with trails. The original port (Phase 2.9) carried 5 hex color constants ('#004c94', '#2e89ff', '#003994', '#004bad', '#ff5900') authored as sRGB by the original CodePen artist. They were passed straight to new THREE.Color(hex) and bound as uColor uniforms. With LinearSRGBColorSpace the green channel of #ff5900 rode into the shader at 3.5× its correct linear value (0.349 instead of 0.099); the compensating overbright was buried in CONFIG.brightness = 4.0131 and inflated hints.bloomHint calibration. The math was wrong; the output happened to look approximately right because of accidental tuning. Phase 3 (2026-05-22) corrected this — trails now reads better than the empirically-tuned version because the natural character of the scene comes through once the underlying math is correct.

The canonical pattern — copy this for every new three scene
// sRGB-authored hex (artist-readable, e.g. from a CodePen / Figma)
const COLOR_HEX = '#ff5900';

// At material construction — decode sRGB to linear at uniform-bind time:
const uColor = new THREE.Color(COLOR_HEX).convertSRGBToLinear();

// Inside the material:
material.uniforms.uColor = { value: uColor };

// Equivalent (Three.js r152+ idiomatic form):
const uColorAlt = new THREE.Color().setHex(0xff5900, THREE.SRGBColorSpace);

// If you author colors as linear values directly (rare — designers think in sRGB):
const uColorLinear = new THREE.Color(1.0, 0.099, 0.0);  // no conversion needed

Use the same pattern for material.color on MeshBasicMaterial / MeshStandardMaterial / etc:

const mat = new THREE.MeshBasicMaterial({
  color: new THREE.Color('#ff5900').convertSRGBToLinear(),
});
What this does NOT apply to
Why this matters now (and not in classic Three.js workflows)

In a "default" Three.js setup (outputColorSpace: SRGBColorSpace, no custom render targets), Three.js handles sRGB → linear → light-math → sRGB conversion transparently in the standard shader chunks. Authors pass sRGB hex, get sRGB-correct output, never think about it.

We're not in that setup. We use LinearSRGBColorSpace because the renderer writes to a HalfFloat target the pipeline tonemaps later — the renderer must NOT pre-encode to sRGB (that would double-encode after the pipeline's ACES + linear-to-sRGB). The price is that the automatic color-space handling Three.js does for material colors is also off — every sRGB color the scene declares needs explicit .convertSRGBToLinear() to land correctly in the linear render path.

See src/render/scenes/trails.js:281 for the working pattern (new THREE.Color(CONFIG[\color${colorIdx}`]).convertSRGBToLinear()`). New three scenes should match this idiom.

category: 'webgpu' — WebGPU compute (8 scenes)

Scenes: compute-chromosphere, compute-fluid, compute-gravity, compute-membrane, compute-particles, compute-spectra, compute-tesla-orb (Plasma Globe), compute-tesseract. (compute-grid and compute-smoke are dev/test entry points — IKANDY_COMPUTE_GRID() / IKANDY_COMPUTE_SMOKE() — not user-facing scenes.)

This is no longer speculative. WebGPU compute is a first-class category with its own runtime stack:

Two authoring paths. A scene factory factory(runtimeApi) → instance returns either a declarative instance (no frame(), but declares storageBuffers + computePasses + renderPass — the orchestrator runs the compute-pass→swap→render-pass chain for you) or an imperative instance (implements frame(ctx) and records/submits its own encoders by hand). Prefer declarative; reach for imperative only when the pass graph is irregular (compute-tesla-orb and compute-fluid are imperative; the rest are declarative).

Lifecycle (parallels every other category's init → SceneInstance outer shape):

register(name, factory)                     ← module load
  ↓
setActive(name)                             ← user picks scene (or harness)
  → bufferPool = new StorageBufferPool(device)
  → instance = await factory(runtimeApi)    // sync OR async; runtime awaits
     runtimeApi = { device, queue, format, context, canvas, width, height,
                    audio, bufferPool, audioUBO, pipelines }
  ↓
loop (per frame):
  instance.update?(audio, dt, frameIndex)   // optional CPU-side per-frame logic
  instance.frame(ctx)                       // REQUIRED — record GPU work
     ctx = { device, queue, view, format, width, height,
             time, dt, frameIndex, audio, bufferPool, audioUBO, pipelines }
  ↓
resize(w, h) → instance.resize?(w, h)
  ↓
clearActive() / setActive(other)
  → instance.dispose?()                     // free buffers/textures/pipelines;
                                            //   do NOT destroy the device
  → bufferPool.dispose()

audio on runtimeApi/ctx is the same live object each frame — read it inside frame()/update(); never cache values across frames. ctx.audio.kick is the runtime-derived hybrid kick (Section 7).

Declarative pattern — copy compute-particles.js:

export function createMyComputeScene(/* api */) {
  return {
    id: 'my-compute', name: 'My Compute', credit: 'Author',
    category: 'webgpu', schema: 2,
    hints: { selfTonemapped: true, audioReactive: true },
    audio: { bass: true, mid: true, treble: true, time: true },

    // Ping-pong storage buffers; the pool allocates 'name' (front) + 'name:out' (back).
    storageBuffers: [
      { name: 'pos', count: 'particleCount', stride: 16, usage: ['storage'],
        pingPong: true, init: 'random-cube' },
      { name: 'vel', count: 'particleCount', stride: 16, usage: ['storage'],
        pingPong: true, init: 'zero' },
    ],

    // Compute pass(es): read front ('pos'/'vel'), write back ('pos:out'/'vel:out').
    computePasses: [
      { name: 'integrate', order: 0, workgroupSize: 64,
        dispatch: { x: 'particleCount' },
        storageBuffers: ['pos', 'vel', 'pos:out', 'vel:out'],
        shader: PHYS_WGSL },
    ],

    // Render pass: reads buffers POST-swap (fresh), draws into #wgpu-canvas.
    renderPass: {
      shader: RENDER_WGSL, topology: 'triangle-list',
      vertexCount: 6, instanceCount: 'particleCount',
      storageBuffers: ['pos', 'vel'],
      blend: 'additive', clear: [0, 0, 0, 1],
    },
  };
}

Imperative pattern — copy compute-tesla-orb.js when the graph is irregular:

init(runtimeApi) {
  // allocate buffers/pipelines here (or let the scene own them directly)
  return {
    frame(ctx) {
      const { device, queue, view, audio, audioUBO } = ctx;
      // CPU-side per-frame work: read audio.kick / audio.bass etc., write scene UBO
      // ... beat-onset detection, physics impulses, control packing ...
      const enc = device.createCommandEncoder();
      const cpass = enc.beginComputePass(); /* dispatch */ cpass.end();
      const rpass = enc.beginRenderPass({ colorAttachments: [{ view, /* ... */ }] });
      /* draw */ rpass.end();
      queue.submit([enc.finish()]);   // imperative scenes own submission
    },
    resize(w, h) { /* reallocate size-dependent targets */ },
    dispose() { /* destroy your buffers/textures/pipelines — NOT the device */ },
  };
}

How the output reaches the screen. Compute scenes do not write into runtime.hdrTarget. They render to #wgpu-canvas (a screen-blended overlay), and PipelineRuntime.setWgpuSourceCanvas(canvas, selfTonemapped, renderFn, edgeTaper) registers that canvas as the pipeline's source — the post-FX chain (bloom, CA, LUT, grain, vignette…) then runs over it each tick, just as it does for the FBO categories. Two args matter for compute authors:

Resource ownership. dispose() must free every GPUBuffer / GPUTexture / pipeline the scene created, but must never call device.destroy() — the device is shared and owned by ComputeRuntime for the lifetime of the runtime (same rule as "don't call renderer.dispose()" for three scenes). The per-scene StorageBufferPool is disposed by the runtime on clearActive.

Color hazard. Compute scenes commonly composite with blend: 'additive'. Additive blending into a non-tonemapped 8-bit target silently clamps to white — see Section 6's v8 additive-blend hazard and the per-channel diagnostic before shipping any layered-additive compute scene.

Reference scenes: compute-particles.js (canonical declarative scene), compute-tesla-orb.js (canonical imperative scene — see Section 11 for why it's the heavily-iterated reference point).

category: 'milkdrop' — not a pipeline category

Butterchurn owns its own preset lifecycle, render loop, post-FX, and DOM canvas (#viz + #mdpp-canvas). It runs as a peer to the pipeline, not under it. See CLAUDE.md → Scene engines for the full architecture.


Section 6 — Color space

Two output modes, declared by hints.selfTonemapped.

selfTonemapped: true (17 of 18 scenes)

Scene writes sRGB-encoded [0, 1] pixels into the HDR FBO. The pipeline's tonemap pass (position 3) is bypassed; output pass (position 9) bypasses linearToSRGB(). Effectively: what the scene draws is what reaches the screen, after intermediate passes (bloom, LUT, grain, CA, scanlines, vignette).

Use this when the scene was authored to look right at full brightness in sRGB — i.e., it has its own tonemap baked in, or it's a low-dynamic-range design (canvas2d).

All 13 fragment scenes (ferrofluid included — the 3D-mesh exception, §11) and all 4 canvas2d scenes set this true (13 + 4 = 17). The fragment scenes were ported from ShaderToy / CodePen sources that author for sRGB display; canvas2d output is inherently sRGB.

selfTonemapped: false (or absent) (1 of 18 scenes — trails)

Scene writes linear-light HDR values into the FBO. Bright pixels are allowed above 1.0 (bloom uses this headroom). The pipeline's ACES tonemap pass curves them down to display range, then the output pass applies sRGB encoding.

Use this for new physically-based scenes that want the pipeline's bloom + tonemap to handle their highlights correctly.

Why bloom looks different across the two modes: the bloom pass runs before tonemap (position 0). It thresholds at 1.0 by default. For selfTonemapped: true scenes, bloom is thresholding already-tonemapped sRGB values — so most pixels are below 1.0 and only the brightest get bloomed. For selfTonemapped: false scenes, bloom is thresholding linear values — so HDR highlights well above 1.0 get bloomed naturally. Per-scene bloomHint.threshold adjustments compensate. (Phase 8.1 documented these per-scene; Phase 2.2 spot-check confirmed the library reads coherent.)

The v8 additive-blend hazard (compute / layered-additive scenes)

8-bit additive blending into a non-tonemapped target silently clamps to white. When a scene composites many bright contributions with blend: 'additive' (one, one, add) into an 8-bit (u8) render target that the pipeline does not tonemap, overlapping hues sum past 255 in each channel and clamp to (255, 255, 255). The result looks lit, animated, and audio-reactive — but every overlapping element has collapsed into a white blob, destroying all hue information. This is the v8 hazard (named for the Tesla Orb v8 regression: the shockwave's intended rainbow burst read as a featureless white ring).

The failure is insidious because it passes every coarse check: pixels are non-zero, the frame changes with audio, brightness tracks the beat. Only the color is wrong. Mitigations live in the scene's WGSL, not the runtime:

Recommended verification — the per-color-channel diagnostic. A pixel-count or mean-diff smoke test cannot catch this (a white blob has plenty of non-zero, audio-varying pixels). Use the pattern in src/test/compute-tesla-orb-diag.html: render the scene, force the worst-case strike frame (max bolts + kick), read the frame back from the GPU, and classify every pixel by per-channel ratios — count "bright", "white" (all channels > 200), and the scene's signature hue (e.g. magenta = R & B high, G much lower). A regression of this class shows up as the signature-hue pixel count collapsing toward zero while "white" climbs. Any compute scene using layered additive composition should pass a per-channel check like this before shipping (Section 10).


Section 7 — Standard uniforms / audio surface

The audio surface is documented in detail in scene-schema.md § AudioFrame and § Uniform contract. Quick reference:

Kick + the audio debug surface (WebGPU compute path)

The compute path derives a beat signal in AudioUBO and exposes a debug probe via ComputeRuntime.getAudioDebug() (console: IKANDY_COMPUTE_RUNTIME.getAudioDebug(), delegates to AudioUBO.getDebug()). The fields it surfaces:

Field Meaning
bass RAW detection signal (pre-user-slider, clamped [0,1]) — what the kick detector runs on
bassVisual the visual bass value scenes actually render (post-slider)
mid / treble visual band values
mainKick Path 1 — transient (rectified bass rise above its slow EMA) OR beat-onset cadence during sustained-loud sections (≤145 BPM, ×0.6 weight)
subKick Path 2 — sharp bass-derivative hits on their own latch (~600 BPM ceiling, capped 0.7) for EDM fills / drum-roll subdivisions
kick combinedmax(mainKick, subKick); the value exposed to scenes as audio.kick
energy, lo, midBand, hi mean of all / low / mid / high thirds of the 32-bin FFT
window min/max/peak of bass/mid/kick/energy since the previous getDebug() call (the right instrument for catching a brief kick spike a one-shot read misses)

Dual-mode kick detection contract. kick is the max of two independent detectors with separate refractory latches, so they can't lock each other out: mainKick catches musical beats (clean transients + sustained-section cadence), subKick catches fast subdivisions. Either path firing wins. The detector runs on the raw band (bassRaw), never the visual band — a hot bass-gain slider (e.g. 2.0) must not shift the detector's operating point or break beat timing.

What scenes should read. Read audio.kick for beats, and audio.bass / audio.mid / audio.treble for band amplitude (these are the visual, slider-scaled values). bassRaw / midRaw / trebleRaw are detection-internal — fed to the kick detector, not for scenes to render. Don't re-implement beat detection off the raw bands in a scene; consume the runtime-derived audio.kick.

Fragment scenes consume all of these via standard uniform names (u_bass, u_time, u_beat, etc.). Canvas2d/three scenes read them from the audio object passed to update().

Declare what you read in the audio: field of the scene metadata. The runtime delivers every field regardless, but the declaration tells the harness what to synthesize during capture and tells the UI which gain sliders are load-bearing.


Section 8 — Resource ownership

The simplest way to stay compliant: every gl.create*() / new THREE.*() / document.createElement('canvas') in init() must have a matching cleanup in dispose().

Resource Allocated by Disposed by
WebGL context (gl) pipeline runtime pipeline runtime
HDR FBO + texture pipeline runtime pipeline runtime
GLSL program (fragment scene) glsl-adapter via helpers.makeProgram() glsl-adapter on dispose()
Image texture (fragment scene) glsl-adapter when scene declares textures: {…} glsl-adapter on dispose()
Schema:2 control state pipeline runtime pipeline runtime
Offscreen canvas + GPU texture (canvas2d) createCanvas2dBridge() bridge.dispose()
THREE.WebGLRenderer three-runtime, lazily on first three scene activation three-runtime on pipeline dispose
Shared WebGLRenderTarget (three) three-runtime via ensureSceneTarget() three-runtime on pipeline dispose
THREE.Scene / Camera / geometries / materials the scene's init() the scene's dispose()
Custom render targets (e.g. trails' floor blur) the scene's init() the scene's dispose()
setTimeout / setInterval / event listeners the scene's init() the scene's dispose()

The ~16-context cap is no longer at risk under the pipeline (one shared context). But texture and buffer counts still matter — a scene that allocates a 10K-particle position texture in init() and forgets to dispose it accumulates ~1MB per scene activation. Over a day of scene cycling that adds up.


Section 9 — Audio-reactivity enforcement

Section 0 says every scene must be audio-reactive. This section says how that's checked — automatically, before merge, for every scene.

Existing mechanism: harness audio-reactive check

src/test/visual-harness.js already runs a silent-vs-loud pixel-diff per scene:

This is the enforcement mechanism. A scene that ships without satisfying it is a bug — either the scene isn't actually reactive (rejected from the library) or the test is wrong (fix the test). Both are blockers.

Look at any recent npm run test:visual log: each scene's report includes a line like v audio-reactive: diff(silent,audio)=49.38 (passing scenes show v with the actual diff magnitude). The check has been live since Phase 2.12.

Wire-band sweeps (additional reactivity coverage)

The harness also runs per-band sweeps (wire-bass, wire-mid, wire-treble, wire-intensity, wire-volume) that vary each audio band min→max with the others held constant. A scene that responds only to bass but declares audio: { mid: true } will fail the wire-mid check even if it passes the overall audio-reactive check — catches mis-declared audio surfaces.

Pass requires per-band pixel-diff ≥ WIRE_AUDIO_MIN (default 1.5).

Visible-effect smoke test (compute scenes — complement to pixel-diff)

Pixel-diff alone is insufficient for compute scenes. A flat white blob passes the silent-vs-loud diff (and the wire sweeps) as easily as a correct render — it has plenty of non-zero, audio-varying pixels while being visually broken (the v8 hazard, Section 6). The diff confirms something changed, not that the right thing changed.

The complement is a visible-effect smoke test that drives real control levers through the live ComputeRuntime and asserts the rendered frame changes in the expected way — see src/test/compute-audio-controls-page.html. It runs each lever and checks a magnitude:

It runs across both authoring paths (imperative Tesla Orb + declarative Particles / Gravity) so the contract — not just "did anything render" — is what's verified. Treat this as the compute-scene analogue of the fragment-scene audio-reactive check: pixel-diff proves reactivity exists; the visible-effect test proves the output is the intended visual rather than a degenerate blob. Pair it with the per-channel diagnostic (Section 6) for any layered-additive scene.

v1.x extensions to the reactivity gate

The existing harness covers the curated 18 scenes well. v1.x extensibility work needs richer enforcement:

All four extensions are documented in the v1.x backlog. None block the v1 launch — the curated library is fully verified by the existing harness check.


Section 10 — Validation checklist for new scenes

Before merging a new scene, confirm:

WebGPU compute scenes (category: 'webgpu') additionally:


Section 11 — Open issues and category exceptions

This section lists deviations from the canonical pattern. Each entry says whether it's a permanent exception or an open issue with a planned resolution.

Stochastic scenes (pawrticles, fire) — open issue, special verification needed

These scenes use Math.random() for per-frame particle state (ember spawn positions, ripple-burst seeds, etc.). The harness currently sets hints.stochasticRender: true on them, which skips the pixel-diff comparison against the reference PNG — because two identical runs produce different per-pixel output, so any reference comparison hits noise.

This is an escape hatch from the pixel-diff method, NOT from the music-visualizer requirement. The scenes might be reactive — the simple pixel-diff just can't prove it. The wire-band sweeps + the audio-reactive check still run against them, and currently pass (recent harness output: fire audio-reactive=85.31, pawrticles audio-reactive=172.23). So they do satisfy Section 0 today.

The gap is that we don't have a robust verification method for stochastic scenes — a scene could be reactive in non-stochastic content (bar heights, overall brightness) while the stochastic layer (embers, particles) drowns the diff in noise. v1.x reactivity work needs one of:

None is implemented yet. For v1 launch, the existing wire + simple-diff checks are good enough — stochastic scenes pass them. The richer method is v1.x.

Ferrofluid — permanent exception (3D mesh in category: 'fragment')

src/render/scenes/ferrofluid.js declares category: 'fragment' but does not use createGlslScene. Instead it builds its own GL pipeline directly: two compiled programs (background fullscreen quad + foreground 3D mesh), a 256×256 vertex grid with indexed draws, a depth renderbuffer attached to the pipeline FBO at init and detached at dispose, per-vertex sampling of a 128-bin R8 audio texture, and mouse drag-to-rotate event listeners on runtime.canvas. Seven gl.create* calls live in init().

Why the bypass is correct: the GLSL adapter is designed for fullscreen- quad fragment shaders. Ferrofluid is a mesh visual — 3D geometry with depth testing, two cooperating programs, and per-vertex (not per-fragment) audio data. None of that fits the adapter's surface. The scene file's header comment (lines 1–32) documents the rationale explicitly.

Why category: 'fragment' instead of inventing 'mesh': the category is a semantic best-fit until we have a second mesh-style scene. The scene still produces sRGB pixels into the standard pipeline path with selfTonemapped: true, so the post-FX chain treats it identically to a fragment scene. Inventing a single-instance 'mesh' category + createMeshScene adapter for one scene is over-engineering.

Resource cleanup is still compliant. dispose() deletes both programs, both buffers, the audio texture, the depth renderbuffer attachment, and removes the canvas event listeners. Audit this when changing the scene.

Trigger condition for revisiting: when (if) a second mesh-style scene is proposed, that's the point to formalize. Pattern: invent category: 'mesh', extract the shared mesh-runtime bits from ferrofluid's init into a createMeshScene adapter (mirroring createGlslScene and createCanvas2dBridge), migrate ferrofluid as the reference scene, then the new scene plugs into the same adapter. Until then, ferrofluid is a one-off and the bypass stays documented here.

MilkDrop — permanent exception (not a category)

Butterchurn owns its own preset lifecycle, render loop, post-FX, and DOM canvas (#viz + #mdpp-canvas). It runs as a peer system to the pipeline, not a category under it. Future MilkDrop integration extensions (mdpp post-FX, texture injection) belong in src/mdpp/* and main/*, not in this contract.

The reactivity gate still applies to MilkDrop presets — they ARE music visualizer content. The v1.x test:capture:milkdrop work is the enforcement mechanism there.

WebGPU compute (category: 'webgpu') — built (was: reserved)

No longer an open item. The ComputeRuntime stack is live and 8 compute scenes ship on it — see Section 5's category: 'webgpu' subsection for the full contract. This entry now only tracks the residual deviations.

Legacy WGSL scenes — open issue, planned unification. Three older WGSL scenes (nebula, ikandy-singularity, ikandy-aurora) still run their own device + start/stop loops on #wgpu-canvas, not through ComputeRuntime. Their loop and the ComputeRuntime loop are mutually exclusive (the host page stops one before starting the other), so reconfiguring the canvas context per setActive is safe. Folding them onto the single shared device/context owner is the planned unification goal (compute-runtime.js header, "Day-7"). Until then they bypass the harness/visible-effect gates because they aren't ComputeRuntime scenes — the reactivity requirement still applies to them in principle.

Plasma Globe (compute-tesla-orb) — canonical heavily-iterated compute scene

src/render/scenes/compute-tesla-orb.js (id: 'compute-tesla-orb', name: 'Plasma Globe', category: 'webgpu', imperative frame(ctx)) is the reference point for a heavily-iterated compute scene — shipped through ~v9 with behavior that goes well past the declarative template. Look here when authoring a compute scene that needs custom reactivity:

It's also the scene that motivated the v8 additive-blend hazard documentation (Section 6) and its per-channel diagnostic (compute-tesla-orb-diag.html). When in doubt about how far a compute scene may deviate while staying compliant, this is the high-water mark: imperative encoding, bespoke detection, and physics are all fine — resource cleanup (Section 8), the color contract (Section 6), and the reactivity gates (Section 9) still bind.


Section 11.5 — Post-FX pass authoring requirements

Different scope from scene authoring — this section is for contributors writing new entries in src/render/passes/*.js (Bloom, CA, Vignette, etc.). Captured here because post-FX passes and scenes interact through the same runtime contract and bugs in one class affect the other.

Visibility floor

Every post-FX pass MUST produce a CLEARLY visible effect at slMul=1.0 on at least one reference scene. Subtle is fine; invisible is a bug.

The perceptual minimums for "user can register that the slider does something":

Authoring checklist

What NOT to do

v1.x extension

When the import / AI authoring paths land, user-uploaded and AI-generated post-FX passes (if we allow them — currently out of scope) must satisfy this section. The harness post-FX visibility check is the enforcement.


Section 12 — Future contract evolution

When v1.x extensibility work (import templates, AI generation) begins, this is the target. Specifically:

Compliance audits should re-run after each major contract change.

← Back to the Artist Program