feat(vision): rehearse a closed loop from a recording - #139
Merged
Conversation
A closed-loop experiment has a lot of links -- pose inference, the behavior model, the node watching for a behavior, the canvas wiring, the device binding, the peripheral -- and any one of them can be wrong in a way that looks fine until an animal is in the box. This plays a recording through the live path instead of the camera, so the whole chain runs on footage where you already know what the animal did. The hardware fires for real, which is the point: a rehearsal that faked the output would not tell you the stimulator works. It needed almost nothing new downstream. _handle_frame_input is a single funnel that every frame already goes through, so a pump emitting _frame_received with the camera's (frame, timestamp) shape gets classification, node triggers and BLE writes unchanged. It emits that signal rather than calling _on_frame, deliberately skipping the _preview_active guard: a rehearsal must not require an open camera, since most machines running one have none. Frames are never skipped. compute_features computes velocity and acceleration over unit frame spacing, so a dropped frame doubles the apparent displacement across the gap and inflates exactly the kinematics the model keys on -- a pump that skipped to keep up would report confident, wrong behavior. It reports its worst lag instead, which is also the number worth reading: a run that ends a second behind will miss stimulus timing on a live animal by the same margin. Both speeds classify identically, for the same reason: unit frame spacing means the playback rate cannot change what the model sees. They answer different questions. Real time answers whether inference keeps up and what the end-to-end latency is; as-fast-as-possible answers whether any of it is wired up, sooner. Offering only the first would have made the wiring check needlessly slow, and offering only the second would have made the latency question unanswerable. A consumer that raises ends the run rather than being swallowed, because continuing would deliver a gap-riddled stream -- the one thing this must never produce.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #138 (retarget to
mainonce that lands — it carries the plugin-registry test fixes this branch needs).Camera → Live behavior → Rehearse from video… plays a recording through the live path instead of the camera. The classifier runs, the nodes fire, and the hardware is driven for real, on footage where you already know what the animal did.
It needed almost nothing new downstream
_handle_frame_inputis a single funnel every frame already passes through:So a pump emitting
_frame_receivedwith the camera's(frame, timestamp)shape — same wall-clocktime.time()— gets the entire live path unchanged. It emits that signal rather than calling_on_frame, deliberately skipping its_preview_activeguard: a rehearsal must not require an open camera, since most machines running one have none.Frames are never skipped
compute_featurescomputes velocity and acceleration over unit frame spacing. A dropped frame doubles the apparent displacement across the gap and inflates exactly the kinematics the behavior model keys on — a pump that skipped to keep up would produce confident, wrong behavior.So it reports its worst lag instead, which is the number worth reading: a run that ends a second behind will miss stimulus timing on a live animal by the same margin.
Both speeds classify identically
Same reason — unit frame spacing means playback rate cannot change what the model sees. They answer different questions:
Offering only the first would make the wiring check needlessly slow; only the second would make the latency question unanswerable.
Tests — 20 new
test_video_pump.py(16) — every frame delivered in order; no frame skipped even when the consumer is slow; lag reported rather than caught up by skipping; no lag reported when unpaced; wall-clock timestamps matching the camera; real-time pacing actually taking the clip's length; unpaced being much faster; stop taking effect mid-clip (it drives hardware, so stop must mean stop); unreadable file failing to start; double-start refused; stop-before-start safe; a raising consumer ending the run rather than gapping it; source released.test_camera_panel_rehearsal.py(4) — a rehearsal frame goes down the camera funnel, not a side channel; the frame is copied (the decoder reuses buffers); it works with no camera open; and a recorded freeze drives a real GATT write end to end, with only pose/behavior inference and bleak faked.Docs
docs-site/camera-behavior/rehearsal.md— how to run one, how to read the lag number, and an explicit what it does and doesn't prove: it won't validate camera setup or lighting, and inference speed only transfers if the clip was recorded on the same camera at the same settings.Verification
PYTHONPATH=src QT_QPA_PLATFORM=offscreen pytest tests/— 3627 passed, 4 skipped, ruff and black clean.