3D Intelligence

Overview

The laboratory works with point clouds, multi-view images, and fully unposed RGB captures. Its systems reason over 3D scenes, object geometry, scene layouts, spatial relationships, open-vocabulary objects, and geometric representations.

M:3L researchers co-authored Z3D, Zoo3D, and TUN3D. These projects address complementary parts of scene-level 3D understanding: language-conditioned grounding, open-vocabulary object detection, and joint layout-plus-object understanding without requiring known camera poses or externally provided depth in the unposed-image setting.

The research supports a path from perception to structured engineering representations: images or point clouds can inform spatial understanding, geometric reconstruction, editable CAD, simulation-ready models, and downstream optimization.

Z3D: Zero-Shot 3D Visual Grounding from Images

Z3D localizes the 3D object referred to by a natural-language query. It operates on multi-view imagery and can optionally use camera poses and depth maps, while its zero-shot formulation avoids task-specific geometric supervision and predefined object priors.

The method combines proposal generation, view selection, prompt-based segmentation, 2D-to-3D lifting, and multi-view aggregation. The associated paper reports state-of-the-art performance among zero-shot methods on ScanRefer and Nr3D at publication time.

• Benchmarks: ScanRefer and Nr3D
• Inputs: point clouds or multi-view images, with posed and unposed-image settings
• Evidence: overall ScanRefer [email protected]/[email protected] of 54.2/46.0 in the zero-shot point-cloud setting; Nr3D overall top-1 accuracy of 54.8

Zoo3D: Zero-Shot 3D Object Detection at Scene Level

Zoo3D introduces a training-free route to scene-level 3D detection. It clusters 2D instance masks across views to construct class-agnostic 3D boxes, then assigns open-vocabulary labels using best-view selection and view-consensus visual-language matching.

Zoo3D0 is zero-shot and requires no training; Zoo3D1 trains a class-agnostic detector on pseudo-labels generated by Zoo3D0. The associated paper reports state-of-the-art open-vocabulary results across ScanNet-based and ARKitScenes evaluations at publication time.

• Benchmarks: ScanNet20, ScanNet60, ScanNet200, and ARKitScenes
• Inputs: point clouds, posed multi-view images, or unposed images
• Evidence: ScanNet200 point-cloud [email protected]/[email protected] of 21.1/14.1 for Zoo3D0 and 23.5/15.2 for Zoo3D1

TUN3D: Towards Real-World Scene Understanding from Unposed Images

TUN3D jointly estimates room layouts and detects 3D objects. In its fully unposed setting, the input is an ordinary image collection without known camera intrinsics, extrinsics, ground-truth poses, or externally provided depth.

DUSt3R reconstructs geometry and camera parameters, while a lightweight sparse-convolutional model uses dedicated object-detection and parametric wall-layout heads. This formulation targets consumer-device capture and recorded imagery rather than depth-sensor-only acquisition.

• Benchmarks: ScanNet, S3DIS, ARKitScenes, and Structured3D
• Unposed ScanNet evidence: layout F1 46.5, detection [email protected] 44.0, and [email protected] 20.7
• Relevant domains: robotics, architecture, building reconstruction, digital twins, embodied AI, and autonomous engineering

Connected research

• CAD Intelligence: M:3L develops multimodal systems that reconstruct structured, editable, executable engineering representations from point clouds, multi-view images, and text.
• Agentic AI: M:3L develops autonomous, tool-using single-agent and multi-agent AI systems for scientific, engineering, and enterprise workflows.