feat(vision): rehearse a closed loop from a recording

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Rehearse a closed loop from a recording.** **Camera → Live behavior → Rehearse from video…** plays a clip through the live path instead of the camera: the classifier runs, the nodes fire, and the hardware is driven for real, on footage where you already know what the animal did. Frames are never skipped — the live feature extractor uses unit frame spacing, so a dropped frame inflates exactly the kinematics the model keys on — so a run that cannot keep up reports its worst lag instead, which is also the number that says whether the rig will hold up live. Real-time and as-fast-as-possible modes classify identically (`compute_features` never reads fps); they differ only in whether they can answer "does inference keep up?".
 - **Maimu moves out of core into an installable plugin, `glider-maimu`.** The device and node are unchanged; what changed is that they are no longer bundled. One lab's stimulator does not belong in every install, and moving it made it the first real consumer of the plugin node extension points — which is the only way to find out whether they are good enough. Two gaps surfaced doing it and are fixed here: a plugin node could not declare its canvas ports (the editor now reads them off the node's own `NodeDefinition` instead of falling back to a generic one-in-one-out, which also fixes every built-in node missing from the hand-written table), and a plugin node could not be bound to a device (a node now declares `REQUIRES_DEVICE`, which `HardwareNode` sets, instead of the editor matching a hardcoded list of type names).
 - **Maimu BLE stimulator** — a `Maimu` device type with the peripheral's GATT layout built in (**Add Device → Maimu → Scan**, no UUIDs to paste), and a **Maimu** node in the library's I/O section offering Mode (On / Off / Pulse) with a period and duration, instead of a Device Action node writing `"500,10"` by hand. Unlike the generic BLE devices, its shutdown writes `off` before disconnecting — the firmware runs a pulse autonomously, so an emergency stop that only dropped the link would leave the device stimulating.
 - **DeepLabCut and SLEAP pose models** — a `PoseBackend` seam behind the three places GLIDER loaded a pose net lets DeepLabCut single-animal and SLEAP single-instance models drive live camera inference, offline classification, and batch runs. They run through `onnxruntime` (new `glider[pose-onnx]` extra), after a one-time export performed in your own DeepLabCut or SLEAP environment with `tools/export_pose_onnx.py` — GLIDER never imports `deeplabcut` or `sleap`, which is what keeps it installable on Python 3.11-3.13 and on the Pi. Ultralytics YOLO `.pt` weights are unchanged: that path is a pass-through over the same code as before. Multi-animal architectures, including SLEAP top-down and bottom-up, are rejected by name rather than silently scoring one arbitrary animal.

docs-site/camera-behavior/rehearsal.md

@@ -0,0 +1,76 @@
+# Rehearsing a closed loop from a recording
+
+A closed-loop experiment — stimulate *because* the animal did something — has a
+lot of links: pose inference, the behavior model, the node that watches for a
+behavior, the wiring on the canvas, the device binding, and the hardware itself.
+Any one of them can be wrong in a way that looks fine until an animal is in the
+box.
+
+**Rehearsal mode plays a recording through the live path instead of the camera.**
+The classifier runs, the nodes fire, and the hardware is driven for real — on
+footage where you already know what the animal did.
+
+!!! warning "The hardware really fires"
+    This is the point: a rehearsal that faked the output would not tell you the
+    stimulator works. Don't rehearse with an animal connected.
+
+## Running one
+
+1. Load your pose model and behavior model in **Camera → Live behavior**, set the
+   keypoint names, and press **Start**. The classifier has to be running — a
+   rehearsal with nothing listening does nothing, and the panel says so.
+2. Build the flow you'd run for real: a **Behavior Input** node watching the
+   behavior you care about, wired to whatever it triggers.
+3. Press **Rehearse from video…** and pick a clip.
+
+The status line reports progress, and at the end it tells you the worst lag.
+
+## Real time or as fast as possible
+
+Both classify the recording **identically**. Feature values come from
+`compute_features`, which uses unit frame spacing and never reads fps — so the
+playback rate cannot change what the model sees. The two modes exist because
+they answer different questions:
+
+| Mode | Answers |
+| --- | --- |
+| **Real time** | Does inference keep up on this machine, and how long does a stimulus take to arrive after a behavior starts? |
+| **As fast as possible** | Is any of this wired up correctly? Same answer, sooner. |
+
+Use *as fast as possible* while you're still fixing wiring, and *real time* for
+the run you actually believe.
+
+## Reading the result
+
+**"kept up with real time"** — inference is fast enough on this machine for this
+footage. The rig should behave the same live.
+
+**"worst lag N ms — inference did not keep up"** — the model is slower than the
+frame rate. A stimulus will arrive late on a live animal by roughly that margin,
+on top of the confirmation delay from the Behavior Input node's `min_frames`
+setting (at 30 fps, 5 frames is about 167 ms). Either is fine if it is small
+against the behavior you are studying, and neither is fine if it isn't.
+
+Frames are **never skipped**, whatever happens. The live feature extractor
+computes velocity and acceleration over unit frame spacing, so a dropped frame
+doubles the apparent displacement across the gap and inflates the kinematics the
+model keys on. A rehearsal that dropped frames to keep up would report confident,
+wrong behavior — so it reports lag instead.
+
+## What it does and doesn't prove
+
+**Does:** the model loads and classifies; `min_frames` is tuned for your actual
+footage; node wiring; the device binds; the peripheral connects; the commands
+sent are the ones you meant; end-to-end latency.
+
+**Doesn't:** camera setup, lighting, or whether inference keeps up on your
+*camera's* resolution and frame rate — unless the clip was recorded on the same
+camera at the same settings. It's worth using footage from the rig you're about
+to run.
+
+## Getting a clip
+
+Any recording the model was trained to handle. A session recorded by GLIDER's own
+video recorder is ideal, because it matches the camera and settings you'll use.
+Pick one where the behavior you're triggering on actually occurs — a rehearsal on
+footage with no freezing tells you nothing about a freeze trigger.

mkdocs.yml

@@ -103,6 +103,7 @@ nav:
       - Tracking: camera-behavior/tracking.md
       - Pose Models: camera-behavior/pose-models.md
       - Behavior Analysis: camera-behavior/behavior.md
+      - Closed-loop Rehearsal: camera-behavior/rehearsal.md
   - Runner Mode:
       - runner/index.md
       - The Runner Screen: runner/runner.md

src/glider/gui/panels/camera_panel.py

@@ -306,6 +306,8 @@ class CameraPanel(QWidget):
     # into the BehaviorInferenceWorker's thread via automatic QueuedConnections.
     _behavior_init_requested = pyqtSignal(str, str, list)  # pkl, pt, keypoint names
     _behavior_frame_requested = pyqtSignal(object)  # FrameData -> worker.process_frame
+    # PumpStats from the rehearsal pump's thread -> the GUI thread.
+    _rehearsal_finished = pyqtSignal(object)
 
     def __init__(
         self,
@@ -345,6 +347,7 @@ def __init__(
         self._behavior_thread: QThread | None = None
         self._behavior_worker: Any | None = None
         self._behavior_running = False
+        self._rehearsal_pump: Any | None = None
 
         # Video-file source state (offline tracking)
         from glider.vision.video_source import VideoFileSource
@@ -598,6 +601,44 @@ def _setup_ui(self) -> None:
         self._live_behavior_btn.clicked.connect(self._toggle_live_behavior)
         behavior_layout.addWidget(self._live_behavior_btn)
 
+        # --- rehearsal: drive the live path from a recording ------------------
+        rehearsal_line = QFrame()
+        rehearsal_line.setFrameShape(QFrame.Shape.HLine)
+        rehearsal_line.setProperty("textRole", "muted")
+        behavior_layout.addWidget(rehearsal_line)
+
+        rehearsal_row = QHBoxLayout()
+        self._rehearse_btn = QPushButton("Rehearse from video\u2026")
+        self._rehearse_btn.setEnabled(False)
+        self._rehearse_btn.setToolTip(
+            "Play a recording through the live path instead of the camera.\n"
+            "Classification, node triggers and hardware all run for real."
+        )
+        self._rehearse_btn.clicked.connect(self._toggle_rehearsal)
+        rehearsal_row.addWidget(self._rehearse_btn, 1)
+
+        self._rehearse_speed = QComboBox()
+        # Feature values do not depend on playback rate -- compute_features uses
+        # unit frame spacing -- so both modes classify identically. They answer
+        # different questions, which is why both exist.
+        self._rehearse_speed.addItem("Real time", 1.0)
+        self._rehearse_speed.addItem("As fast as possible", 0.0)
+        self._rehearse_speed.setToolTip(
+            "Real time answers 'does inference keep up, and what is the "
+            "latency?'.\nAs fast as possible answers 'is this wired up?' "
+            "sooner.\nBoth classify the recording identically."
+        )
+        rehearsal_row.addWidget(self._rehearse_speed)
+        behavior_layout.addLayout(rehearsal_row)
+
+        self._rehearse_status = QLabel(
+            "Start live behavior first \u2014 a rehearsal with no classifier "
+            "running does nothing."
+        )
+        self._rehearse_status.setProperty("textRole", "muted")
+        self._rehearse_status.setWordWrap(True)
+        behavior_layout.addWidget(self._rehearse_status)
+
         layout.addWidget(behavior_group)
 
         # Set up scroll area
@@ -615,6 +656,7 @@ def _connect_signals(self) -> None:
 
         # Connect thread-safe signals for UI updates (main thread)
         self._frame_received.connect(self._handle_frame_input)
+        self._rehearsal_finished.connect(self._on_rehearsal_finished_main)
         self._multi_frame_received.connect(self._handle_multi_frame_input)
 
         # Connect CV worker signals
@@ -646,6 +688,88 @@ def _connect_signals(self) -> None:
         self._run_fps = FpsMeter()
         self._run_frames_done = 0
 
+    # --- rehearsal from a recording ---------------------------------------
+
+    def _toggle_rehearsal(self) -> None:
+        if self._rehearsal_pump is not None and self._rehearsal_pump.is_running:
+            self._stop_rehearsal()
+        else:
+            self._start_rehearsal()
+
+    def _start_rehearsal(self) -> None:
+        """Play a recording through the live path, hardware and all."""
+        from PyQt6.QtWidgets import QFileDialog
+
+        from glider.vision.video_pump import VideoPump
+
+        path, _ = QFileDialog.getOpenFileName(
+            self, "Rehearse from video", "", "Videos (*.mp4 *.avi *.mov *.mkv);;All Files (*)"
+        )
+        if not path:
+            return
+
+        speed = self._rehearse_speed.currentData()
+        self._rehearsal_pump = VideoPump(
+            path,
+            self._on_rehearsal_frame,
+            speed=speed,
+            on_finished=self._on_rehearsal_finished,
+        )
+        if not self._rehearsal_pump.start():
+            self._rehearsal_pump = None
+            self._rehearse_status.setText(f"Could not open {Path(path).name}.")
+            return
+
+        self._rehearse_btn.setText("Stop rehearsal")
+        self._rehearse_speed.setEnabled(False)
+        self._rehearse_status.setText(
+            f"Rehearsing {Path(path).name} \u2014 hardware is being driven for real."
+        )
+        logger.info("Rehearsal started from %s", path)
+
+    def _stop_rehearsal(self) -> None:
+        pump = self._rehearsal_pump
+        if pump is not None:
+            pump.stop()
+        self._rehearsal_pump = None
+        self._rehearse_btn.setText("Rehearse from video\u2026")
+        self._rehearse_speed.setEnabled(True)
+        logger.info("Rehearsal stopped")
+
+    def _on_rehearsal_frame(self, frame: np.ndarray, timestamp: float) -> None:
+        """Hand a recorded frame to the live path.
+
+        Called from the pump's thread, exactly as ``_on_frame`` is called from
+        the camera's capture thread, and doing the same thing: copy, then emit
+        so the work happens on the GUI thread. Emitting ``_frame_received``
+        rather than calling ``_on_frame`` deliberately skips its
+        ``_preview_active`` guard -- a rehearsal does not need a camera open,
+        and requiring one would mean it could not run on a machine that has
+        none.
+        """
+        self._frame_received.emit(FrameData(frame=frame.copy(), timestamp=timestamp))
+
+    def _on_rehearsal_finished(self, stats: Any) -> None:
+        """Report how it went. Called from the pump's thread."""
+        self._rehearsal_finished.emit(stats)
+
+    def _on_rehearsal_finished_main(self, stats: Any) -> None:
+        self._rehearse_btn.setText("Rehearse from video\u2026")
+        self._rehearse_speed.setEnabled(True)
+        self._rehearsal_pump = None
+
+        lag_ms = stats.max_lag_s * 1000
+        summary = f"Rehearsal finished: {stats.frames_delivered} frames"
+        if stats.max_lag_s > 0:
+            # The number that says whether the rig will hold up live: a run
+            # that ended a second behind will miss stimulus timing on an animal
+            # by the same margin.
+            summary += f", worst lag {lag_ms:.0f} ms \u2014 inference did not keep up"
+        elif self._rehearse_speed.currentData():
+            summary += ", kept up with real time"
+        self._rehearse_status.setText(summary)
+        logger.info("%s", summary)
+
     def _handle_frame_input(self, frame_data: FrameData) -> None:
         """Decide whether to process frame with CV or update UI immediately."""
         # Fan out EVERY frame to the live-behavior worker (independent of the CV
@@ -1602,6 +1726,8 @@ def _on_behavior_ready(self) -> None:
         if worker is None:
             return
         self._preview.set_behavior_vocab(worker.classes)
+        self._rehearse_btn.setEnabled(True)
+        self._rehearse_status.setText("Ready to rehearse from a recording.")
         # Also hand the vocabulary to the flow side, so a Behavior Input node's
         # properties can offer the behaviors this model actually emits instead
         # of a free-text box where a typo means "never fires".
@@ -1711,6 +1837,9 @@ def closeEvent(self, event):
         # Tear down any in-flight tracking run and release the scrub video source.
         self._teardown_run_thread()
         self.stop_live_behavior()
+        if self._rehearsal_pump is not None:
+            self._rehearsal_pump.stop()
+            self._rehearsal_pump = None
         self._video_source.release()
 
         if self._preview_active: