diff --git a/docs/internal/video/README.md b/docs/internal/video/README.md index 1889e6b3f..d528e75d0 100644 --- a/docs/internal/video/README.md +++ b/docs/internal/video/README.md @@ -64,17 +64,17 @@ What the recorders do and why: - **CDP screencast, not `recordVideo`.** Playwright's video is VP8 at about 1 Mbit/s: blurry UI text. The screencast gives a JPEG (quality 92) per repaint with its timestamp; `index.json` lists frames and named marks (`start`, `act`, `start:fish`) to cut on. - **Size.** Feature clips record 1280x720 CSS pixels at device scale 1.5: 1920x1080 frames with the UI 1.5x larger than at 1080p, so it stays readable when the stage frames a part of it. Split-screen pages record at about their panel size. The screencast sends device pixels only when the browser itself runs at the page's scale, so `launch()` starts one Chromium per scale with `--force-device-scale-factor`; with the page's `deviceScaleFactor` alone a 1.5x page comes as 1280x720 frames. - **Recording-only CSS** is fine: the Snake page is stacked into a column and its address box hidden (`addStyleTag`); nothing changes in the product. -- **Few frames are fine.** A page that does not change sends no frames; the stage holds the last one and dissolves a jump (an unfold, a scroll) over 0.3 s instead of cutting. +- **Few frames are fine.** A page that does not change sends no frames; the stage holds the last one. It does not dissolve between frames: the dashboard's own motion is what shows (a dissolve after a pause was cut short by the next frame and read as a jerk). Gotchas we hit: - Several servers from one worktree build the dashboard at the same time into the same `static/` and crash; start one, wait for its build, then the rest. -- **Warm up before the take**, so the Flow keeps still while it is recorded: bring every agent onto it (`external` appears with the first artifact; Snake publishes one step itself in a run of its own, Quick Draw draws one picture so the guesser's card has its thumbnails), fit it once, switch Auto-arrange off, then start recording. A fit or an auto-arrange during the take moves the whole board. +- **Warm up before the take**, so the Flow keeps still while it is recorded: bring every agent onto it (`external` appears with the first artifact; Snake publishes one step itself in a run of its own, Quick Draw draws one picture so the guesser's card has its thumbnails), fit it once, then start recording. A fit during the take moves the whole board. Snake switches Auto-arrange off (its three agents never change size); Quick, Draw! keeps it on, so the guesser's card, which grows with every guess, stays in the middle of the Flow (#655). - The Blackboard shows the time range only: the examples publish slowly, so switch to the last hour (`lastHour`) before looking for a decision's lane. - Unfolded decision cards are taller than a 720 px window: scroll the section that matters (Escalation, Vote, Shadow, Explanation) to the middle. - `azure/decision-1` allows 100 requests per minute (`decision_rate_limit="100/min"` in the demo): Snake plays about 100 steps, then stalls until the window frees. 20 s of play is the most one take gives. - A vision model may guess the house from its first stroke; draw every stroke anyway, so the picture is finished on screen. -- Frame times come from a monotonic clock (`record()` notes when each frame arrives). The screencast's own timestamps follow the wall clock, which on WSL steps back by about 2 s every 30 s: the stage would play those frames out of order. `check_motion.py` says when a recording's times step back. +- Frame times are the compositor's (when Chromium painted the frame: frames of a glide are exactly 16.7 ms apart) moved onto a monotonic clock. Arrival times jitter by a few ms, and a glide sampled at 30 fps from them stutters; the compositor's times follow the wall clock, which on WSL steps back (by up to 2 s every 30 s, sometimes by tens of ms): `record()` measures the wall clock against the monotonic clock at each frame, and never places a frame after its arrival (a step between paint and delivery) or without a time (CDP may leave it out). `check_motion.py` says when a recording's times still step back. - A board that keeps changing moves its lanes on purpose (a new run arrives on top, a stream ends and its card gets shorter): these glide. Pick the calmest seconds with `check_motion.py --calm` rather than cutting around every glide. - Stop servers by PID (`ss -ltnp | grep :PORT`, `kill -9 PID`), never `pkill -f` with a pattern that is also in your own command line. @@ -84,7 +84,7 @@ Gotchas we hit: - **Scenes:** `scene(name, seconds, build)`; `build(root, seconds)` creates the scene's elements once and returns `update(t)` with the scene's own time. Scenes overlap by 0.5 s and crossfade; titles leave before their scene ends. - **Format:** `SHAPE` is `wide` (wider than 1.3:1), `square` or `tall`; `U` is one unit of type and spacing (the short side / 1080); `M` the margin. `twoPanels()` puts two windows side by side unless the frame is tall, where it stacks them by shares of its height and crops them: a clip's `tall` camera (`tall: { cx, cy, zoom }`) picks what shows, e.g. the Snake canvas (`cy: 0.3`). The feature reel puts its text left of the window when wide and below it otherwise. Write every position from `W`, `H`, `U` and `M`, never in fixed pixels. -- **Windows:** `panel()` films a recording through a camera: `show(name, time, zoom, cx, cy)` centres (cx, cy) of the recording (0..1) at the given zoom over a cover fit, never past its edges. Frame each feature by its centre and zoom; push in slowly (a few percent over the clip). +- **Windows:** `panel()` films a recording through a camera: `show(name, time, zoom, cx, cy)` centres (cx, cy) of the recording (0..1) at the given zoom over a cover fit, never past its edges. Frame each feature by its centre and zoom, and keep the camera still: a slow zoom rescales every frame a little and makes the UI shimmer. The reel's windows are about 1.2:1, narrower than a 16:9 recording: frame the part that tells the feature (a section of a card, the bracket, a lane's head), and check every shot in a preview at the middle and the end of its window. - **Type:** `title(parent, 'FLOCK CAN NOW *PLAY!', size)`: words rise in one after another; `*word` is amber, `~word` blue. `chip(parent, head, sub, size)`: a small card with a line of code that slides in and out. - **Content:** `ANCHORS` lists the feature clips (recording, verb, code, line, camera x, camera y, zoom); `VERSION` is the intro's version. Recordings are referred to by name (`snake-game`, `clip-escalate`, ...). diff --git a/docs/internal/video/stage/stage.js b/docs/internal/video/stage/stage.js index 1d07001cc..2fc431669 100644 --- a/docs/internal/video/stage/stage.js +++ b/docs/internal/video/stage/stage.js @@ -109,24 +109,22 @@ async function load(name) { } const mark = (name, m) => recordings[name].marks[m]; -// The frame of `name` shown at absolute time `abs`; a repaint after a pause -// (an unfold, a jump) dissolves in over 0.3 s instead of cutting +// The frame of `name` shown at absolute time `abs`: the last one painted by +// then. No dissolves: the dashboard's own motion is what shows (a dissolve +// after a pause was cut short by the next frame and read as a jerk) function frameAt(name, abs) { const { frames } = recordings[name]; if (abs <= frames[0].t) return { cur: frames[0], prev: null, a: 1 }; let lo = 0, hi = frames.length - 1; while (lo < hi) { const mid = (lo + hi + 1) >> 1; if (frames[mid].t <= abs) lo = mid; else hi = mid - 1; } - const cur = frames[lo], prev = frames[lo - 1]; - if (prev && cur.t - prev.t > 0.12) { - const a = clamp((abs - cur.t) / 0.3); - if (a < 1) return { cur, prev, a: inOutCubic(a) }; - } - return { cur, prev: null, a: 1 }; + return { cur: frames[lo], prev: null, a: 1 }; } const pending = new Set(); // A window that shows a recording through a camera: `zoom` over the cover -// fit, centred on (cx, cy) of the recording (0..1), never past its edges +// fit, centred on (cx, cy) of the recording (0..1), never past its edges. The +// camera keeps still while a recording plays: a slow zoom rescales every +// frame a little and makes the UI shimmer function panel(parent, { radius = 16 } = {}) { const e = el('div', 'panel', parent); e.style.borderRadius = `${radius}px`; @@ -271,8 +269,7 @@ twoWindowScene('play', 18, { aspects: [528 / 880, 1280 / 860], clips: [ { rec: 'snake-game', at: (t) => mark('snake-game', 'start') - 2.4 + t, tall: { cy: 0.3 } }, - // The Flow is fitted to the board's width before the take: a slight push-in keeps every agent in view - { rec: 'snake-board', at: (t) => mark('snake-board', 'start') - 2.4 + t, zoom: (t) => lerp(1, 1.06, inOutCubic(seg(t, 4, 11))), cy: 0.5 }, + { rec: 'snake-board', at: (t) => mark('snake-board', 'start') - 2.4 + t, cy: 0.5 }, ], chips: [ { head: 'Decision models', sub: 'calibrated probabilities · azure/decision-1', at: 4.2, out: 8.2, dot: 'blue' }, @@ -294,7 +291,8 @@ twoWindowScene('see', 19, { aspects: [1080 / 700, 520 / 860], clips: [ { rec: 'draw-app', at: drawAt('draw-app'), tall: { cx: 0.45 } }, - { rec: 'draw-board', at: drawAt('draw-board'), zoom: (t) => lerp(1.25, 1.4, seg(t, 3, 19)), cy: 0.46 }, + // Full height: the guesser grows with every guess, Auto-arrange keeps it in the middle + { rec: 'draw-board', at: drawAt('draw-board'), zoom: () => 1, cy: 0.5 }, ], chipSide: 'a', chips: [ @@ -324,8 +322,8 @@ scene('dashboard', 8, (root) => { head.words(t, 0.1, 7.2); const a = outExpo(seg(t, 0.3, 1.1)); set(before.e, { o: a, y: (1 - a) * 60 * U }); - before.show('clip-before', mark('clip-before', 'act') + 0.5 + t, lerp(1, 1.05, seg(t, 0, 8))); - after.show('clip-after', mark('clip-after', 'act') + t - 2, lerp(1.05, 1, seg(t, 2, 8))); + before.show('clip-before', mark('clip-before', 'act') + 0.5 + t); + after.show('clip-after', mark('clip-after', 'act') + t - 2); const wipe = inOutCubic(seg(t, 3.0, 4.6)); // the new one sweeps in from the right after.e.style.clipPath = `inset(0 0 0 ${(1 - wipe) * 100}%)`; set(after.e, { o: a }); @@ -341,16 +339,16 @@ scene('dashboard', 8, (root) => { // 5 ALSO: one anchor per feature; the camera frames the feature in each recording // [recording, verb (last word tinted), code, line, camera x, camera y, zoom] const ANCHORS = [ - ['clip-escalate', 'ESCALATE', 'escalate=["openai/gpt-6-luna"]', 'not sure? the next model answers', 0.53, 0.52, 1.45], - ['clip-vote', 'VOTE', 'models=[…], vote="majority"', 'three models, one decision', 0.53, 0.56, 1.45], - ['clip-shadow', 'SHADOW', 'shadow="openai/gpt-6-luna"', 'try a model without routing on it', 0.53, 0.5, 1.45], + ['clip-escalate', 'ESCALATE', 'escalate=["openai/gpt-6-luna"]', 'not sure? the next model answers', 0.58, 0.52, 1.45], + ['clip-vote', 'VOTE', 'models=[…], vote="majority"', 'three models, one decision', 0.58, 0.56, 1.45], + ['clip-shadow', 'SHADOW', 'shadow="openai/gpt-6-luna"', 'try a model without routing on it', 0.58, 0.5, 1.45], ['clip-explain', 'EXPLAIN ITSELF', '.explains(model=…)', 'an LLM writes why', 0.58, 0.42, 1.3], - ['clip-tournament', 'RUN TOURNAMENTS', 'Tournament(group_size=20, keep=3)', 'hundreds of options, in rounds', 0.27, 0.45, 1.3], + ['clip-tournament', 'RUN TOURNAMENTS', 'Tournament(group_size=20, keep=3)', 'hundreds of options, in rounds', 0.25, 0.42, 1.4], ['clip-checklist', 'TICK CHECKLISTS', 'Checklist: one yes/no per item', 'with tokens and cost per run', 0.6, 0.48, 1.35], - ['clip-coverage', 'ROLL UP COVERAGE', '.aggregates(by="organisation")', 'checklists across documents', 0.71, 0.45, 1.25], + ['clip-coverage', 'ROLL UP COVERAGE', '.aggregates(by="organisation")', 'checklists across documents', 0.69, 0.48, 1.08], ['clip-review', 'ASK A HUMAN', 'flock.review(decision_id, choice=…)', 'unsure decisions wait in a queue', 0.63, 0.55, 1.35], - ['clip-lanes', 'SHOW EVERY RUN', 'the Blackboard as run lanes', 'live, one lane per correlation', 0.33, 0.45, 1.25], - ['clip-focus', 'FOCUS', 'double-click any agent', 'lineage, live output, runs, traces', 0.42, 0.42, 1.2], + ['clip-lanes', 'SHOW EVERY RUN', 'the Blackboard as run lanes', 'live, one lane per correlation', 0.3, 0.45, 1.1], + ['clip-focus', 'FOCUS', 'double-click any agent', 'lineage, live output, runs, traces', 0.36, 0.45, 1.0], ['clip-palette', 'FIND ANYTHING', 'Ctrl / ⌘ + K', 'agents, types, runs and commands', 0.5, 0.22, 1.3], ['clip-theme', 'CHANGE ITS LOOK', 'Graphite · Paper · terminal themes', 'and every shortcut in one dialog', 0.6, 0.5, 1.3], ]; @@ -392,7 +390,7 @@ scene('also', LEAD + ANCHORS.length * PER, (root, dur) => { const enter = outExpo(seg(lt, -0.4, 0.5)); set(it.p.e, { o: seg(lt, -0.4, 0.0), x: (1 - enter) * 120 * U, s: lerp(1.03, 1, enter) }); it.p.e.style.zIndex = 1 + i; - it.p.show(it.rec, mark(it.rec, 'act') - 0.4 + lt, it.zoom * lerp(0.94, 1.04, seg(lt, 0, PER + 0.5)), it.cx, it.cy); + it.p.show(it.rec, mark(it.rec, 'act') - 0.4 + lt, it.zoom, it.cx, it.cy); // The text block: kicker, verb, chip, centred in the column (below the window when not wide) const vs = it.v.size(), cs = it.c.size(); const block = 40 * U + vs.h + 26 * U + cs.h; diff --git a/docs/internal/video/tools/lib.mjs b/docs/internal/video/tools/lib.mjs index 150b1f199..d954214f5 100644 --- a/docs/internal/video/tools/lib.mjs +++ b/docs/internal/video/tools/lib.mjs @@ -19,6 +19,7 @@ export const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); */ const now = () => (performance.timeOrigin + performance.now()) / 1000; + /** * Chromium, one browser per device scale: the screencast sends frames at device * pixels only when the browser itself runs at the page's scale @@ -51,10 +52,15 @@ export async function newPage(browser, width, height, scale = 1, colorScheme = ' /** * Screencast `page` into FRAMES/-raw: one JPEG per repaint and index.json - * with each frame's time (seconds since the epoch, monotonic: when the frame - * arrived). mark(name) notes a moment to cut on. + * with each frame's time in seconds since the epoch on the monotonic clock. + * The time is the compositor's (when the frame was painted: exact, frames of a + * glide 16.7 ms apart) moved onto the monotonic clock by the wall clock's skew + * against it, measured at each frame (a wall-clock step changes the skew, a + * late delivery does not); arrival times jitter by a few ms, which made + * glides stutter when the stage sampled them at 30 fps. mark(name) notes a + * moment to cut on. * Pages repaint only when something changes, so a still page gives few frames; - * the stage holds the last frame and dissolves jumps. + * the stage holds the last frame. */ export async function record(page, name, { quality = 92 } = {}) { const dir = path.join(FRAMES, `${name}-raw`); @@ -64,8 +70,15 @@ export async function record(page, name, { quality = 92 } = {}) { const frames = []; const marks = []; let n = 0; - client.on('Page.screencastFrame', async ({ data, sessionId }) => { - const t = now(); + let last = 0; + client.on('Page.screencastFrame', async ({ data, metadata, sessionId }) => { + const received = now(); + const skew = Date.now() / 1000 - received; // wall clock minus monotonic, now + // CDP may leave the timestamp out; a step between the paint and its delivery + // moves `painted` by the step: a frame is not painted after it arrives, nor a second before + const painted = Number.isFinite(metadata.timestamp) ? metadata.timestamp - skew : received; + const t = Math.max(Math.min(painted, received), received - 1, last + 0.001); + last = t; const file = `f${String(n++).padStart(6, '0')}.jpg`; fs.writeFileSync(path.join(dir, file), Buffer.from(data, 'base64')); frames.push({ file, t }); diff --git a/docs/internal/video/tools/record-draw.mjs b/docs/internal/video/tools/record-draw.mjs index 93caec4ed..635a8a9d6 100644 --- a/docs/internal/video/tools/record-draw.mjs +++ b/docs/internal/video/tools/record-draw.mjs @@ -24,8 +24,9 @@ async function drawable(done = []) { } // Warm-up, not recorded: every agent and the guesser's thumbnails are on the -// board, then the Flow is fitted once and keeps still (no auto-arrange), so -// the card only grows during the take instead of jumping and zooming +// board, then the Flow is fitted once. Auto-arrange stays on: a card that +// grows lays the board out again, centred and gliding (#655), so the guesser +// grows up and down instead of out of the view { const prompt = await drawable(); const box = await app.locator('#canvas').boundingBox(); @@ -34,7 +35,7 @@ async function drawable(done = []) { await board.locator('[aria-label="Fit to screen"]').click(); await sleep(1200); const arrange = board.locator('button[role=switch]:has-text("Auto-arrange")'); - if ((await arrange.getAttribute('aria-checked')) === 'true') await arrange.click(); + if ((await arrange.getAttribute('aria-checked')) !== 'true') await arrange.click(); await app.locator('#new').click(); await sleep(1500); } diff --git a/docs/internal/video/tools/record-features.mjs b/docs/internal/video/tools/record-features.mjs index 28aa504cc..a3e994d96 100644 --- a/docs/internal/video/tools/record-features.mjs +++ b/docs/internal/video/tools/record-features.mjs @@ -23,8 +23,8 @@ async function clip(name, prepare, act, { seconds = 6, ready = 'text=Live' } = { await sleep(1200); const rec = await record(page, `clip-${name}`); await sleep(900); - rec.mark('act'); // the stage cuts each clip from this moment - await act(page, ctx); + rec.mark('act'); // the stage cuts each clip from this moment (an act may mark it again, later) + await act(page, ctx, rec); await sleep(seconds * 1000); console.log(name, 'frames', await rec.stop()); await page.context().close(); @@ -34,7 +34,7 @@ async function clip(name, prepare, act, { seconds = 6, ready = 'text=Live' } = { // that just started may publish one only after a few minutes: wait for it const laneOf = (port, test, label = null) => async (page) => { let candidates = []; - for (const until = Date.now() + 240_000; !candidates.length && Date.now() < until; ) { + for (const until = performance.now() + 240_000; !candidates.length && performance.now() < until; ) { candidates = decisionsOf(await snapshot(port)).filter(test).reverse(); if (!candidates.length) await sleep(5000); } @@ -69,11 +69,22 @@ await clip('escalate', laneOf(PORTS.escalate, (d) => d.tier === 1 && d.choice != await clip('vote', laneOf(PORTS.vote, (d) => d.choice !== 'UNSURE' && d.votes?.length >= 3), unfold(/^vote$/i)); await clip('shadow', laneOf(PORTS.shadow, (d) => d.shadow && !d.shadow.error && d.shadow.choice !== d.choice), unfold(/^shadow$/i)); await clip('explain', laneOf(PORTS.explain, (d) => d.explanation), unfold(/^explanation$/i)); -await clip('tournament', laneOf(PORTS.tournament, (d) => d.rounds?.length, 'by tournament'), async (page, ctx) => { - await unfold()(page, ctx); - await sleep(1500); +// The bracket is open before the take; the take traces options through it (hover) +await clip('tournament', async (page) => { + const ctx = await laneOf(PORTS.tournament, (d) => d.rounds?.length, 'by tournament')(page); + await ctx.card.click(); + await sleep(1000); await ctx.lane.locator('article[class*="unfolded"]').first().getByRole('button', { name: /bracket/i }).first().click(); -}, { seconds: 7 }); + await sleep(1200); + return ctx; +}, async (page) => { + // The winner's path from its first round to the final, then another finalist's + const option = (filter) => page.locator(`[data-testid="bracket-option"][data-column="0"]${filter}`).first(); + await option('[data-winner="true"]').hover(); + await sleep(1600); + await option('[data-survivor="true"][data-winner="false"]').hover(); + await sleep(1200); +}, { seconds: 2 }); await clip('checklist', async (page) => { await lastHour(page); await fit(page); }, async (page) => { await page.locator('.react-flow__node[data-id="audit"]').click(); // unfold: answers, runs, tokens }); @@ -88,8 +99,18 @@ await clip('review', async (page) => { await lastHour(page); await fit(page); }, // The dashboard (the triage demo) // Long: the board moves as runs arrive; the stage takes the calmest seconds (check_motion.py --calm) await clip('lanes', async () => {}, async (page) => { await blackboard(page); }, { seconds: 16 }); -await clip('focus', async (page) => { await fit(page); }, async (page) => { - await page.locator('.react-flow__node[data-id="triage"]').dblclick(); +// An agent that writes (an LLM): the focus view shows its live output +const WRITERS = ['tech_support', 'billing_team', 'shipping_team', 'supervisor']; +// Chosen in the take, not before: a reply streams for a second or two +await clip('focus', async (page) => { await fit(page); }, async (page, _ctx, rec) => { + let node = page.locator(`.react-flow__node[data-id="${WRITERS[0]}"]`); + for (const until = performance.now() + 30_000; performance.now() < until; await sleep(100)) { // monotonic: the wall clock steps on WSL + const running = WRITERS.map((name) => page.locator(`.react-flow__node[data-id="${name}"]`)); + const found = await Promise.all(running.map((each) => each.locator('[data-status="running"]').count())); + if (found.some(Boolean)) { node = running[found.findIndex(Boolean)]; break; } + } + rec.mark('act'); + await node.dblclick(); }); await clip('palette', async (page) => { await fit(page); }, async (page) => { await page.keyboard.press('Control+k');