danvi/dim1486/generalize dan perception stack

[bogwi]

Generealize perception stack

This PR builds, improves, and adds new functionality based on the foundation of these previous PRs:
#3496
#3422

How to test and what to expect

refference machine:
Lenovo Legion laptop, AMD Ruyzen AI 7, 32GB RAM, NVIDIA RTX 5070 Laptop edition (8 GB)

Caller - operator/agent decide the localization parameters at runtime

every localize call can have own tuning parameters supplied

At the moment we have ROOM policy, and default table policy, think xArm gripper.

This accounts for different environment and rig. We can control a detection score to take only the best candidates, and an acceptance floor, you can control segmentation, etc. Check localize.py

xArm

This uses the deault location for xArm dataset, already on main, xarm6_worldbelief_20260729_203624_161992.db.
You do not need to specify the --dataset <location_to_recording_bag_.db>

time uv run python -m dimos.perception.memory.tool_localize "roll of black tape" "book" "pen" "red marker" "yellow sticky notes" pc_xArm_all_multi.rrd --from 427 --duration 76 --multi && uv run dimos-viewer pc_xArm_all_multi.rrd

# cold start, weights ~ 11s reff.machine
real	0m34.002s 
user	1m4.301s
sys	    0m5.702s

G1

You will need a g1 recording for this, check this PR, it's on main already, #3527

time uv run python -m dimos.perception.memory.tool_localize "paper box" "plant in a pot" "shoes" "wooden crate" "office chair" "bag" g1_sf_120_180_localize.rrd --dataset data/g1_sf_office.db --from 120 --duration 60 --multi && uv run dimos-viewer g1_sf_120_180_localize.rrd

# cold start, weights ~ 11s reff.machine
real	1m11.971s
user	4m44.586s
sys	    0m11.864s

Go2

Run on the default go2_short.db

time uv run python -m dimos.perception.memory.tool_localize "table" "white robot" "potted plants" "wooden doors" "coke bottle" "fanta bottle" "sprite bottle" "lays chips" "pillow" "green plant" "plant pot" go2_short_00_60.rrd --dataset data/go2_short.db --from 0 --duration 60 --multi && uv run dimos-viewer go2_short_00_60.rrd

# cold start, weights ~ 11s reff.machine
real    1m4.001s
user    1m45.383s
sys     0m24.279s

Important

0. Performance is affected:

• by a cold run, weights need to be loaded, around ~11 seconds on my machine
• saving to *.rrd format

1. tool_inventory.py has been not touched by this commit, will come later.
   Use localize() API for testing and building
2. Re-id and spatial memory are underway too
3. Improving objects 2D into 3D is also underway
4. supporting mid360 lidar is also undeerway
5. Any questions - contact me directly, @bogwi

[codecov]

Codecov Report

❌ Patch coverage is 48.30508% with 61 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...erception/detection/type/detection3d/pointcloud.py	54.32%	28 Missing and 9 partials ⚠️
...n/detection/type/detection3d/pointcloud_filters.py	5.00%	19 Missing ⚠️
dimos/memory/backend.py	66.66%	1 Missing and 1 partial ⚠️
.../detection/type/detection3d/imageDetections3DPC.py	33.33%	2 Missing ⚠️
dimos/memory/stream.py	0.00%	1 Missing ⚠️

@@            Coverage Diff             @@
##             main    #3723      +/-   ##
==========================================
+ Coverage   77.46%   77.55%   +0.09%     
==========================================
  Files        1302     1315      +13     
  Lines      123061   124789    +1728     
  Branches    10772    11283     +511     
==========================================
+ Hits        95330    96782    +1452     
+ Misses      24924    24864      -60     
- Partials     2807     3143     +336     

Flag	Coverage Δ
OS-ubuntu-24.04-arm	74.18% <14.40%> (+0.69%)	⬆️
OS-ubuntu-latest	74.73% <14.40%> (+0.67%)	⬆️
Py-3.10	74.73% <14.40%> (+0.67%)	⬆️
Py-3.11	74.73% <14.40%> (+0.67%)	⬆️
Py-3.12	74.73% <14.40%> (+0.68%)	⬆️
Py-3.13	74.73% <14.40%> (+0.67%)	⬆️
Py-3.14	74.73% <14.40%> (+0.68%)	⬆️
Py-3.14t	74.73% <14.40%> (+0.68%)	⬆️
SelfHosted-Large	30.20% <10.16%> (-0.14%)	⬇️
SelfHosted-Linux	34.74% <44.91%> (-0.12%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/memory/utils/sqlite.py	100.00% <100.00%> (ø)
...tion/detection/type/detection3d/test_pointcloud.py	100.00% <100.00%> (ø)
dimos/memory/stream.py	83.48% <0.00%> (ø)
dimos/memory/backend.py	86.18% <66.66%> (+0.94%)	⬆️
.../detection/type/detection3d/imageDetections3DPC.py	60.00% <33.33%> (-9.24%)	⬇️
...n/detection/type/detection3d/pointcloud_filters.py	54.71% <5.00%> (-30.14%)	⬇️
...erception/detection/type/detection3d/pointcloud.py	48.24% <54.32%> (+1.62%)	⬆️

... and 155 files with indirect coverage changes

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[greptile-apps]

Greptile Summary

This change broadens perception-memory localization across depth-camera and registered point-cloud rigs, adds persistent object identities, batches detector scoring, and expands recording CLI configuration.

Two localization reliability problems remain. Short embedded recordings can yield no semantic frame to inspect, so objects in one- or two-frame windows are never sent to detection. Separately, a frame that temporarily lacks usable 3D geometry is permanently skipped when the same identity store is reused, even after geometry becomes available.

T-Rex validation blocked

The short-recording check could not run to completion because the Python runtime was missing repository dependencies. The final import failure was for the plum package, so the harness did not reach the localization path.

Confidence Score: 3/5

Not safe to merge without addressing the two localization failure paths, because valid objects can be omitted from results.

One failure was reproduced by executing two localization calls against a reused identity store and demonstrating recovery after delaying ingestion. The short-recording failure is directly supported by the selection path, but its runtime harness was blocked by missing Python packages.

Files Needing Attention: dimos/perception/memory/localize.py needs fallback candidate selection for short semantic indexes and delayed persistence of ingestion state; dimos/memory/transform.py may also need a short-stream behavior that yields a usable result.

T-Rex Logs

What T-Rex did

• T-Rex produced a proof for a posted P1 finding using a focused two-call harness, and the activity is supported by the harness source and runtime outputs.
• Validation was blocked due to a missing Python package (plum) in the authored short-index harness, preventing execution and producing a ModuleNotFoundError.
• T-Rex produced another finding-proof for a posted P1 finding.
• Execution was blocked: the localize-short-index-repro.py script could not run, with baseline failing on numpy and environment issues, and subsequent imports failing for reactivex, pydantic_settings, dimos_lcm, and plum, leaving no executable observation of the localization behavior.

_{Ran code and verified through T-Rex}

_{Reviews (1): Last reviewed commit: "ship go2_short.db.rig.json inside the go..." | Re-trigger Greptile}

[leshy]

basically escaped mem2 immediately, first comments first view to push back on this, might have more stuff