feat(vision): closed-loop behavior triggering — freeze fires the Maimu - #131
Merged
Conversation
Vision produced a result per frame on a worker thread and nodes that react to it live on the asyncio loop, with nothing connecting the two. ZoneInputNode carried an update method that nothing in the codebase ever called, so its exec outputs could never fire; the same hole meant a behavior could be displayed but never act on anything. LiveSignalBus is the missing hop. Nodes subscribe when they start and unsubscribe when they stop, and whichever component is running inference publishes to it without knowing what is listening. The alternative -- having the camera panel reach into the flow and look for nodes by type -- would make the GUI import the node package and re-implement that lookup for every new node type. Delivery is forgiving: a subscriber that raises is logged and skipped, because the publisher is a vision thread whose job is not to care about the flow graph. BehaviorInputNode watches for one behavior and fires On Enter / On Exit as the animal enters and leaves it. Two properties of the live signal shape it. The per-frame label is noisy -- raw frame-wise macro F1 is 0.777 against 0.811 for the same frames under a 25-frame centred vote -- so wiring a raw label straight to hardware means one misclassified frame fires a stimulus; entering requires min_frames consecutive matching frames and leaving the same number without. The centred vote itself cannot be used, because it reads frames that have not happened yet: legitimate offline, impossible live. Consecutive-frame confirmation is the causal equivalent, and its cost is explicit -- at 30 fps a min_frames of 5 delays the trigger by about 167 ms on top of inference. There is no confidence output. The live path's LiveResult carries a label and keypoints only, so a confidence port would read zero on every frame and invite gating on a number that was never measured. BehaviorEvent keeps an optional confidence field for publishers that do have one.
The closed loop has four hops -- classifier frame, LiveSignalBus, Behavior Input node, exec connection, Maimu node, MaimuDevice, GATT write -- and every one of them was built separately. This drives all of them at once and asserts on the bytes that reach the peripheral rather than on an intermediate signal. Only the two genuinely external ends are stubbed: the classifier, which contributes a label per frame exactly as classify_frame yields one, and bleak. Covers the behaviours that matter for a stimulus that fires at an animal: a confirmed freeze pulses; a single stray misclassified frame does not; a sustained freeze pulses once per bout rather than once per frame; leaving and re-entering pulses twice; a stopped flow drives nothing; and the classifier's empty warm-up label counts as a miss rather than as a behaviour. Documents an ordering property found while writing it: FlowEngine.start() schedules each continuous node's start() as a task and returns before it has run, so a node is not on the bus yet when start() returns. Frames published in that window are dropped. Harmless at 30 fps, where the next frame is 33 ms away, but it silently breaks anything that publishes immediately.
Freezing was the worked example and the file name implied it was the feature. It is not: BehaviorInputNode compares the label the classifier emitted against the one it was configured with, so darting, grooming, rearing or anything else the model was trained to recognise drives hardware identically. Nothing in the path is freezing-specific, and nothing should quietly become so. Adds the cases that keep it that way: several behaviors parametrised through the whole chain, a watcher staying silent through a sustained bout of a behavior it was not watching, and two watchers on one bus driving two separate stimulators with their own pulse parameters -- which is the shape a real experiment takes. Renames the file accordingly.
This was referenced Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the loop: the animal freezes, the Maimu stimulates.
Important
Provenance. The first commit is work that was sitting uncommitted in the
.claude/worktrees/behavior-trigger-nodeworktree — another agent session, same author. It is rebased onto currentmainhere, unchanged apart from one conflict resolution. If that session is still live, it should stop and reset rather than continue on a stale base. The second commit is new.What this enables
A stimulus delivered because of what the animal is doing, not because a timer expired.
Commit 1 —
feat(vision): the missing hopVision produced a result per frame on a worker thread; nodes that react to it live on the asyncio loop. Nothing connected the two.
ZoneInputNodecarried an update method that nothing in the codebase ever called, so zone exec outputs could never fire either — this fixes that class of gap, not just the behavior case.LiveSignalBus(core/live_signals.py) — nodes subscribe on start and unsubscribe on stop; whoever runs inference publishes without knowing what listens. A subscriber that raises is logged and skipped, because the publisher is a vision thread whose job is not to care about the flow graph.BehaviorInputNode(nodes/vision/behavior_nodes.py) —Active/Behaviordata outputs plus On Enter / On Exit exec outputs.Two properties of the live signal shape the node, and both are worth reading before tuning it:
min_framesconsecutive matching frames; leaving requires the same number without.min_framesof 5 delays the trigger by ~167 ms on top of inference.No confidence output, deliberately: the live path's
LiveResultcarries a label and keypoints only, so a confidence port would read zero every frame and invite gating on a number that was never measured.Commit 2 —
test(integration): proof, on the bytestests/integration/test_freeze_triggers_maimu.pydrives all four hops at once and asserts on the GATT payload (b"500,10") reaching the peripheral, not on an intermediate signal. Only the two genuinely external ends are stubbed — the classifier (a label per frame, exactly whatclassify_frameyields) and bleak.min_framesexists to prevent""until its feature buffer fillsAn ordering property found while writing it
All three positive tests failed on the first run. Probing each hop rather than guessing:
FlowEngine.start()schedules each continuous node'sstart()as a task and returns before it has run, so a node is not on the bus yet whenstart()returns, and frames published in that window are dropped silently.Harmless in production — at 30 fps the next frame is 33 ms away and the start task runs on the next loop iteration — but it is undocumented and will silently break anything that publishes immediately. Documented in the test rather than papered over. Worth deciding separately whether
start()should await node startup.Verification
PYTHONPATH=src QT_QPA_PLATFORM=offscreen pytest tests/— 3621 passed, 3 skipped, ruff and black clean. The 34 tests from commit 1 pass unchanged on top of the merged Maimu work.Outside the code
For a real closed-loop run: the trained model needs
freezingin itsclasses_, and live inference must be running in the camera panel. The bus only carries what the classifier actually emits.