Skip to content

feat(vision): rehearse a closed loop from a recording - #139

Merged
gbradham merged 1 commit into
fix/maimu-entry-point-namesfrom
feat/video-rehearsal-mode
Aug 21, 2026
Merged

feat(vision): rehearse a closed loop from a recording#139
gbradham merged 1 commit into
fix/maimu-entry-point-namesfrom
feat/video-rehearsal-mode

Conversation

@gbradham

Copy link
Copy Markdown
Member

Stacked on #138 (retarget to main once that lands — it carries the plugin-registry test fixes this branch needs).

Camera → Live behavior → Rehearse from video… plays a recording through the live path instead of the camera. The classifier runs, the nodes fire, and the hardware is driven for real, on footage where you already know what the animal did.

⚠️ The stimulator actually fires. That is the point — a rehearsal that faked the output would not tell you the hardware works. Don't rehearse with an animal connected.

It needed almost nothing new downstream

_handle_frame_input is a single funnel every frame already passes through:

camera loop ─┐
             ├→ _frame_received → _handle_frame_input ─┬→ behavior worker → bus → nodes → hardware
video pump  ─┘                                          └→ CV worker → tracking + overlay

So a pump emitting _frame_received with the camera's (frame, timestamp) shape — same wall-clock time.time() — gets the entire live path unchanged. It emits that signal rather than calling _on_frame, deliberately skipping its _preview_active guard: a rehearsal must not require an open camera, since most machines running one have none.

Frames are never skipped

compute_features computes velocity and acceleration over unit frame spacing. A dropped frame doubles the apparent displacement across the gap and inflates exactly the kinematics the behavior model keys on — a pump that skipped to keep up would produce confident, wrong behavior.

So it reports its worst lag instead, which is the number worth reading: a run that ends a second behind will miss stimulus timing on a live animal by the same margin.

Both speeds classify identically

Same reason — unit frame spacing means playback rate cannot change what the model sees. They answer different questions:

Mode Answers
Real time Does inference keep up, and what is the latency from behavior onset to stimulus?
As fast as possible Is any of this wired up? Same answer, sooner.

Offering only the first would make the wiring check needlessly slow; only the second would make the latency question unanswerable.

Tests — 20 new

test_video_pump.py (16) — every frame delivered in order; no frame skipped even when the consumer is slow; lag reported rather than caught up by skipping; no lag reported when unpaced; wall-clock timestamps matching the camera; real-time pacing actually taking the clip's length; unpaced being much faster; stop taking effect mid-clip (it drives hardware, so stop must mean stop); unreadable file failing to start; double-start refused; stop-before-start safe; a raising consumer ending the run rather than gapping it; source released.

test_camera_panel_rehearsal.py (4) — a rehearsal frame goes down the camera funnel, not a side channel; the frame is copied (the decoder reuses buffers); it works with no camera open; and a recorded freeze drives a real GATT write end to end, with only pose/behavior inference and bleak faked.

Docs

docs-site/camera-behavior/rehearsal.md — how to run one, how to read the lag number, and an explicit what it does and doesn't prove: it won't validate camera setup or lighting, and inference speed only transfers if the clip was recorded on the same camera at the same settings.

Verification

PYTHONPATH=src QT_QPA_PLATFORM=offscreen pytest tests/3627 passed, 4 skipped, ruff and black clean.

A closed-loop experiment has a lot of links -- pose inference, the behavior
model, the node watching for a behavior, the canvas wiring, the device binding,
the peripheral -- and any one of them can be wrong in a way that looks fine
until an animal is in the box. This plays a recording through the live path
instead of the camera, so the whole chain runs on footage where you already
know what the animal did. The hardware fires for real, which is the point: a
rehearsal that faked the output would not tell you the stimulator works.

It needed almost nothing new downstream. _handle_frame_input is a single funnel
that every frame already goes through, so a pump emitting _frame_received with
the camera's (frame, timestamp) shape gets classification, node triggers and
BLE writes unchanged. It emits that signal rather than calling _on_frame,
deliberately skipping the _preview_active guard: a rehearsal must not require
an open camera, since most machines running one have none.

Frames are never skipped. compute_features computes velocity and acceleration
over unit frame spacing, so a dropped frame doubles the apparent displacement
across the gap and inflates exactly the kinematics the model keys on -- a pump
that skipped to keep up would report confident, wrong behavior. It reports its
worst lag instead, which is also the number worth reading: a run that ends a
second behind will miss stimulus timing on a live animal by the same margin.

Both speeds classify identically, for the same reason: unit frame spacing means
the playback rate cannot change what the model sees. They answer different
questions. Real time answers whether inference keeps up and what the end-to-end
latency is; as-fast-as-possible answers whether any of it is wired up, sooner.
Offering only the first would have made the wiring check needlessly slow, and
offering only the second would have made the latency question unanswerable.

A consumer that raises ends the run rather than being swallowed, because
continuing would deliver a gap-riddled stream -- the one thing this must never
produce.
@gbradham
gbradham merged commit f67441e into fix/maimu-entry-point-names Aug 21, 2026
@gbradham
gbradham deleted the feat/video-rehearsal-mode branch August 21, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant