Expo iOS App vm control

[danielehrhardt]

You can now control a VM from the phone app, not just watch it. Tap "Take control" on a running VM's live screen and your taps, scrolls and typing go to the VM. Core tests for the new route pass, and I ran the full flow in the iOS Simulator against a stand-in for the VM. It has not been tried against a real VM yet.

Take control of a VM from the phone app

Until now the phone app's live view could only watch a VM: "Take control" was turned off for VMs because the core had no way to send input to them.

Core

• New route POST /api/vms/:id/input takes { event, frame }, using the same kinds of events as the shared-screen input (click, move, drag, scroll, key, text).
• How it works: a new function, dispatchVmInput in vm/screen.ts, scales the points from the picture the phone shows to the VM's actual screen and sends them over the VM's existing Screen Sharing connection.
  • Scrolling turns into wheel steps.
  • Keys go through the same key-name parser as computer use, so names like "Enter" or "cmd+w" work.
  • Long or non-English text is pasted through the VM's clipboard.
• It never starts a VM. A stopped VM answers 409, a point outside the screen or an unknown key answers 400.
• Phone access: mobile/scope.ts lets phones call the route, and the request may only carry event and frame. Phones can control any of Godmode's VMs, not only screens shared in a chat. docs/ARCHITECTURE.md (Phone app → Scope) says so.

App (apps/mobile)

• Take control for VMs: "Take control" now shows for running VMs. After each input the picture refreshes right away instead of waiting for the next poll (every 1.5 s).
• Right-click: for VMs and shared screens, tap clicks and hold right-clicks. A right-click shows its ripple in blue.
• Scrolling: separate scroll up and scroll down buttons.
• Key bar above the type box: esc, tab, delete and the arrow keys. VMs also get ⌘ space, ⌘ tab, ⌘ W and ⌘ Q.
• Fixes in the typing panel that also apply to browser and shared-screen views:
  • The type box, key bar and buttons used to sit under the on-screen keyboard. They now stay just above it, and the picture moves up into the space that is left.
  • "Enter" sends any text still in the box first, then Return. Before, the text was dropped.
  • Auto-capitalisation and autocorrect are off in the type box, so commands and web addresses go in as typed.

Tests

• New test in vms.test.ts:
  • input to a stopped VM is refused and doesn't start it;
  • a right-click on a 64×48 picture lands at the right spot on the 128×96 test screen;
  • scrolling sends the right number of wheel steps;
  • a key with ⌘ sends the right key events;
  • points off the screen, unknown keys and a missing frame answer 400.
• mobile.test.ts: phones may call the route, and a request carrying any other field is refused.
• bun test test/vms.test.ts test/mobile.test.ts: 52 passed, 0 failed. Typecheck passes for core and for the app.
• The app's lint couldn't run: eslint isn't installed in apps/mobile.

Simulator check

I ran the app from this branch on an iPhone 17 Simulator. A test copy of Godmode supplied a stand-in VM, with a made-up macOS desktop as its screen, and logged what the phone sent. Every control arrived as intended:

• a tap at the right point on the screen;
• hold for a right-click;
• esc, Enter and scrolling;
• typed text followed by Enter, in that order.

Screenshots:

• /Users/danielehrhardt/.godmode/agents/godmode/workspace/task9-shots/vm-control.png: control mode with the scroll buttons and the hint line.
• /Users/danielehrhardt/.godmode/agents/godmode/workspace/task9-shots/vm-typing.png: typing with the on-screen keyboard open; the key bar and type box sit above it and the picture is in full view.

---

Task #9 · done by Godmode with Godmode Bot