Skip to content

feat(vision): closed-loop behavior triggering — freeze fires the Maimu - #131

Merged
gbradham merged 3 commits into
mainfrom
feat/closed-loop-behavior-trigger
Aug 20, 2026
Merged

feat(vision): closed-loop behavior triggering — freeze fires the Maimu#131
gbradham merged 3 commits into
mainfrom
feat/closed-loop-behavior-trigger

Conversation

@gbradham

Copy link
Copy Markdown
Member

Closes the loop: the animal freezes, the Maimu stimulates.

Important

Provenance. The first commit is work that was sitting uncommitted in the .claude/worktrees/behavior-trigger-node worktree — another agent session, same author. It is rebased onto current main here, unchanged apart from one conflict resolution. If that session is still live, it should stop and reset rather than continue on a stale base. The second commit is new.

What this enables

Behavior Input (freezing) ──On Enter──▶ Maimu (Pulse 500 ms / 10 s)

A stimulus delivered because of what the animal is doing, not because a timer expired.

Commit 1 — feat(vision): the missing hop

Vision produced a result per frame on a worker thread; nodes that react to it live on the asyncio loop. Nothing connected the two. ZoneInputNode carried an update method that nothing in the codebase ever called, so zone exec outputs could never fire either — this fixes that class of gap, not just the behavior case.

  • LiveSignalBus (core/live_signals.py) — nodes subscribe on start and unsubscribe on stop; whoever runs inference publishes without knowing what listens. A subscriber that raises is logged and skipped, because the publisher is a vision thread whose job is not to care about the flow graph.
  • BehaviorInputNode (nodes/vision/behavior_nodes.py) — Active / Behavior data outputs plus On Enter / On Exit exec outputs.

Two properties of the live signal shape the node, and both are worth reading before tuning it:

  • The per-frame label is noisy: raw frame-wise macro F1 is 0.777 against 0.811 for the same frames under a 25-frame centred vote. Wiring a raw label straight to hardware means one misclassified frame fires a stimulus. Entering requires min_frames consecutive matching frames; leaving requires the same number without.
  • The centred vote cannot be used here — it reads frames that have not happened yet. Legitimate offline, impossible live. Consecutive-frame confirmation is the causal equivalent, and its cost is explicit: at 30 fps a min_frames of 5 delays the trigger by ~167 ms on top of inference.

No confidence output, deliberately: the live path's LiveResult carries a label and keypoints only, so a confidence port would read zero every frame and invite gating on a number that was never measured.

Commit 2 — test(integration): proof, on the bytes

tests/integration/test_freeze_triggers_maimu.py drives all four hops at once and asserts on the GATT payload (b"500,10") reaching the peripheral, not on an intermediate signal. Only the two genuinely external ends are stubbed — the classifier (a label per frame, exactly what classify_frame yields) and bleak.

Covered Why it matters
Confirmed freeze pulses The headline
One stray frame does not The failure mode min_frames exists to prevent
Sustained freeze pulses once per bout On Enter is an edge, not a level
Leave and re-enter pulses twice Bout detection actually resets
A stopped flow drives nothing Stopping an experiment must stop the loop, not just the display
Empty warm-up label is a miss The classifier emits "" until its feature buffer fills

An ordering property found while writing it

All three positive tests failed on the first run. Probing each hop rather than guessing: FlowEngine.start() schedules each continuous node's start() as a task and returns before it has run, so a node is not on the bus yet when start() returns, and frames published in that window are dropped silently.

Harmless in production — at 30 fps the next frame is 33 ms away and the start task runs on the next loop iteration — but it is undocumented and will silently break anything that publishes immediately. Documented in the test rather than papered over. Worth deciding separately whether start() should await node startup.

Verification

PYTHONPATH=src QT_QPA_PLATFORM=offscreen pytest tests/3621 passed, 3 skipped, ruff and black clean. The 34 tests from commit 1 pass unchanged on top of the merged Maimu work.

Outside the code

For a real closed-loop run: the trained model needs freezing in its classes_, and live inference must be running in the camera panel. The bus only carries what the classifier actually emits.

Vision produced a result per frame on a worker thread and nodes that react
to it live on the asyncio loop, with nothing connecting the two. ZoneInputNode
carried an update method that nothing in the codebase ever called, so its exec
outputs could never fire; the same hole meant a behavior could be displayed
but never act on anything.

LiveSignalBus is the missing hop. Nodes subscribe when they start and
unsubscribe when they stop, and whichever component is running inference
publishes to it without knowing what is listening. The alternative -- having
the camera panel reach into the flow and look for nodes by type -- would make
the GUI import the node package and re-implement that lookup for every new
node type. Delivery is forgiving: a subscriber that raises is logged and
skipped, because the publisher is a vision thread whose job is not to care
about the flow graph.

BehaviorInputNode watches for one behavior and fires On Enter / On Exit as the
animal enters and leaves it. Two properties of the live signal shape it. The
per-frame label is noisy -- raw frame-wise macro F1 is 0.777 against 0.811
for the same frames under a 25-frame centred vote -- so wiring a raw label
straight to hardware means one misclassified frame fires a stimulus; entering
requires min_frames consecutive matching frames and leaving the same number
without. The centred vote itself cannot be used, because it reads frames that
have not happened yet: legitimate offline, impossible live. Consecutive-frame
confirmation is the causal equivalent, and its cost is explicit -- at 30 fps a
min_frames of 5 delays the trigger by about 167 ms on top of inference.

There is no confidence output. The live path's LiveResult carries a label and
keypoints only, so a confidence port would read zero on every frame and invite
gating on a number that was never measured. BehaviorEvent keeps an optional
confidence field for publishers that do have one.
The closed loop has four hops -- classifier frame, LiveSignalBus, Behavior
Input node, exec connection, Maimu node, MaimuDevice, GATT write -- and every
one of them was built separately. This drives all of them at once and asserts
on the bytes that reach the peripheral rather than on an intermediate signal.
Only the two genuinely external ends are stubbed: the classifier, which
contributes a label per frame exactly as classify_frame yields one, and bleak.

Covers the behaviours that matter for a stimulus that fires at an animal: a
confirmed freeze pulses; a single stray misclassified frame does not; a
sustained freeze pulses once per bout rather than once per frame; leaving and
re-entering pulses twice; a stopped flow drives nothing; and the classifier's
empty warm-up label counts as a miss rather than as a behaviour.

Documents an ordering property found while writing it: FlowEngine.start()
schedules each continuous node's start() as a task and returns before it has
run, so a node is not on the bus yet when start() returns. Frames published in
that window are dropped. Harmless at 30 fps, where the next frame is 33 ms
away, but it silently breaks anything that publishes immediately.
Freezing was the worked example and the file name implied it was the feature.
It is not: BehaviorInputNode compares the label the classifier emitted against
the one it was configured with, so darting, grooming, rearing or anything else
the model was trained to recognise drives hardware identically. Nothing in the
path is freezing-specific, and nothing should quietly become so.

Adds the cases that keep it that way: several behaviors parametrised through
the whole chain, a watcher staying silent through a sustained bout of a
behavior it was not watching, and two watchers on one bus driving two separate
stimulators with their own pulse parameters -- which is the shape a real
experiment takes.

Renames the file accordingly.
@gbradham
gbradham merged commit 573fd59 into main Aug 20, 2026
4 checks passed
@gbradham
gbradham deleted the feat/closed-loop-behavior-trigger branch August 20, 2026 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant