feat(g1): add PICO SONIC WebXR teleoperation - #3721
Draft
TomCC7 wants to merge 93 commits into
Draft
Conversation
GEAR-SONIC planner+encoder+decoder ported from the Matrix parity reimplementation into a ControlTask claiming all 29 joints. All 27 locomotion modes RPC-reachable via task_invoke; twist via the standard duck-typed hook; upper-body targets are encoder hints only (never post-decoder overrides). Armature-derived SONIC PD gains carried over exactly, including the doubled ankle/waist entries. Blueprint unitree-g1-sonic-wbc with the standard sim/real dual backend. Verified standalone against raw MuJoCo: 40 s continuous choreography (walk 4 m, squat, boxing, dance) with zero falls and zero NaN at 0.23 ms/step policy latency on CPU onnxruntime.
Groot-style URDF static robot + joint-state overrides, entity key matching zenoh's stripped topic naming (world/g1_joints). Guarded off under DIMOS_TRANSPORT=zenoh: callable factories in rerun_config do not survive zenoh module-deploy serialization (msgpack -> dicts -> validation reject); applies to any blueprint, works under LCM.
SONIC's native protocol as a first-class task input: wire codec for the 1280-byte packed-header format (round-trip verified against NVIDIA's own builders), command/planner/pose topic routing polled non-blocking from compute(), StreamedMotionMerger ported faithfully (sliding window, catch-up, protocol v1/v2/v3, encode-mode selection), SMPL + wrist encoder fields for mode 2, wire-order 17-DoF upper-body targets with velocities, direct planner commands overriding twist mapping, g1_debug state PUB. E2E verified in sim against the live blueprint driven ONLY by NVIDIA's sender code: planner-mode walk, then protocol-v1 streamed-motion playback of the macarena reference clip - 110 planner + 110 pose messages, zero decode errors, zero tick errors, no falls (base z 0.79 at end).
PICO WebXR omits its platform-reserved menu button, so controller packets contain six buttons. Treat the trailing menu slot as optional instead of rejecting otherwise valid joystick input.
# Conflicts: # dimos/teleop/webxr/extensions.py # dimos/teleop/webxr/test_module.py
…eat/pico-teleop-sonic # Conflicts: # dimos/robot/test_all_blueprints.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
unitree-g1-sonic-webxr-teleopblueprint and native MuJoCo visualization pathDependencies
This draft integrates the three prerequisite branches for end-to-end simulation testing.
Validation
uv run dimos --simulation mujoco --viewer none run unitree-g1-sonic-webxr-teleopLocal runtime note
ONNX Runtime falls back to CPU when the CUDA execution provider is unavailable.