# 3D Intelligence

> M:3L develops geometric and spatial AI for zero-shot 3D grounding, open-vocabulary detection, scene layouts, and real-world understanding from point clouds or multi-view imagery.

## Overview

The laboratory works with point clouds, multi-view images, and fully unposed RGB captures. Its systems reason over 3D scenes, object geometry, scene layouts, spatial relationships, open-vocabulary objects, and geometric representations.

M:3L researchers co-authored Z3D, Zoo3D, and TUN3D. These projects address complementary parts of scene-level 3D understanding: language-conditioned grounding, open-vocabulary object detection, and joint layout-plus-object understanding without requiring known camera poses or externally provided depth in the unposed-image setting.

The research supports a path from perception to structured engineering representations: images or point clouds can inform spatial understanding, geometric reconstruction, editable CAD, simulation-ready models, and downstream optimization.

## Demonstrated M:3L research

- Z3D: zero-shot 3D visual grounding from multi-view images, evaluated on ScanRefer and Nr3D.
- Zoo3D: training-free and self-supervised open-vocabulary 3D object detection, evaluated on ScanNet and ARKitScenes.
- TUN3D: joint scene-layout estimation and 3D object detection from point clouds, posed images, or fully unposed RGB imagery, evaluated on ScanNet, S3DIS, ARKitScenes, and Structured3D.

## M:3L technical capabilities

- Zero-shot 3D visual grounding
- Training-free open-vocabulary 3D object detection
- Joint scene-layout estimation and 3D detection
- Unposed multi-view image understanding
- Point-cloud, posed-image, and unposed-image processing
- Geometric reasoning and spatial relationships
- Open-world and open-vocabulary perception

## Z3D: Zero-Shot 3D Visual Grounding from Images

Z3D localizes the 3D object referred to by a natural-language query. It operates on multi-view imagery and can optionally use camera poses and depth maps, while its zero-shot formulation avoids task-specific geometric supervision and predefined object priors.

The method combines proposal generation, view selection, prompt-based segmentation, 2D-to-3D lifting, and multi-view aggregation. The associated paper reports state-of-the-art performance among zero-shot methods on ScanRefer and Nr3D at publication time.

- Benchmarks: ScanRefer and Nr3D
- Inputs: point clouds or multi-view images, with posed and unposed-image settings
- Evidence: overall ScanRefer Acc@0.25/Acc@0.5 of 54.2/46.0 in the zero-shot point-cloud setting; Nr3D overall top-1 accuracy of 54.8

## Zoo3D: Zero-Shot 3D Object Detection at Scene Level

Zoo3D introduces a training-free route to scene-level 3D detection. It clusters 2D instance masks across views to construct class-agnostic 3D boxes, then assigns open-vocabulary labels using best-view selection and view-consensus visual-language matching.

Zoo3D0 is zero-shot and requires no training; Zoo3D1 trains a class-agnostic detector on pseudo-labels generated by Zoo3D0. The associated paper reports state-of-the-art open-vocabulary results across ScanNet-based and ARKitScenes evaluations at publication time.

- Benchmarks: ScanNet20, ScanNet60, ScanNet200, and ARKitScenes
- Inputs: point clouds, posed multi-view images, or unposed images
- Evidence: ScanNet200 point-cloud mAP@0.25/mAP@0.5 of 21.1/14.1 for Zoo3D0 and 23.5/15.2 for Zoo3D1

## TUN3D: Towards Real-World Scene Understanding from Unposed Images

TUN3D jointly estimates room layouts and detects 3D objects. In its fully unposed setting, the input is an ordinary image collection without known camera intrinsics, extrinsics, ground-truth poses, or externally provided depth.

DUSt3R reconstructs geometry and camera parameters, while a lightweight sparse-convolutional model uses dedicated object-detection and parametric wall-layout heads. This formulation targets consumer-device capture and recorded imagery rather than depth-sensor-only acquisition.

- Benchmarks: ScanNet, S3DIS, ARKitScenes, and Structured3D
- Unposed ScanNet evidence: layout F1 46.5, detection mAP@0.25 44.0, and mAP@0.5 20.7
- Relevant domains: robotics, architecture, building reconstruction, digital twins, embodied AI, and autonomous engineering

## Research evidence and benchmarks

### [Z3D: Zero-Shot 3D Visual Grounding from Images](https://m3l.am/publications/z3d.md)

| Benchmark | Task | Input | Metric | Result | Evidence |
| --- | --- | --- | --- | --- | --- |
| ScanRefer | Zero-shot 3D visual grounding | Ground-truth point cloud; no bounding-box or text supervision | Overall Acc@0.25 / Acc@0.5 | 54.2 / 46.0 | Z3D paper, Table 1; Reported as state of the art among zero-shot approaches in the associated paper. |
| ScanRefer | Zero-shot 3D visual grounding | Posed multi-view RGB | Overall Acc@0.25 / Acc@0.5 | 42.8 / 24.8 | Z3D paper, Table 1 |
| ScanRefer | Zero-shot 3D visual grounding | Unposed multi-view RGB | Overall Acc@0.25 / Acc@0.5 | 31.2 / 12.9 | Z3D paper, Table 1 |
| Nr3D | Zero-shot 3D visual grounding | Depth-aware setting | Overall top-1 accuracy | 54.8 | Z3D paper, Table 2; The strongest zero-shot result in the paper's main Nr3D table was 54.8 versus 54.3 for SPAZER. |

### [Zoo3D: Zero-Shot 3D Object Detection at Scene Level](https://m3l.am/publications/zoo3d.md)

| Benchmark | Task | Input | Metric | Result | Evidence |
| --- | --- | --- | --- | --- | --- |
| ScanNet200 | Open-vocabulary 3D object detection (Zoo3D0) | Point cloud and posed scene images; training-free | mAP@0.25 / mAP@0.5 | 21.1 / 14.1 | Zoo3D paper, Table 2 |
| ScanNet200 | Open-vocabulary 3D object detection (Zoo3D1) | Point cloud and posed scene images; self-supervised pseudo-label training | mAP@0.25 / mAP@0.5 | 23.5 / 15.2 | Zoo3D paper, Table 2 |
| ScanNet200 | Open-vocabulary 3D object detection (Zoo3D0 / Zoo3D1) | Fully unposed multi-view images | mAP@0.25 / mAP@0.5 | 8.3 / 2.9 (Zoo3D0); 10.7 / 3.8 (Zoo3D1) | Zoo3D paper, Table 2 |
| ARKitScenes | Open-vocabulary 3D object detection (Zoo3D0 / Zoo3D1) | Point cloud and posed images | mAP@0.25 / mAP@0.5 | 24.4 / 11.0 (Zoo3D0); 34.2 / 24.2 (Zoo3D1) | Zoo3D paper, Table 15 |
| ARKitScenes | Open-vocabulary 3D object detection (Zoo3D0 / Zoo3D1) | Unposed multi-view images | mAP@0.25 / mAP@0.5 | 13.0 / 2.6 (Zoo3D0); 16.1 / 3.5 (Zoo3D1) | Zoo3D paper, Table 15 |

### [TUN3D: Towards Real-World Scene Understanding from Unposed Images](https://m3l.am/publications/tun3d.md)

| Benchmark | Task | Input | Metric | Result | Evidence |
| --- | --- | --- | --- | --- | --- |
| ScanNet | Joint layout estimation and 3D object detection | Unposed multi-view RGB; no known poses or external depth | Layout F1 / detection mAP@0.25 / mAP@0.5 | 46.5 / 44.0 / 20.7 | TUN3D paper, Table I |
| S3DIS | Joint layout estimation and 3D object detection | Unposed multi-view RGB; no known poses or external depth | Layout F1 / detection mAP@0.25 / mAP@0.5 | 20.8 / 11.0 / 2.2 | TUN3D paper, Table I |
| ScanNet | Joint layout estimation and 3D object detection | Posed multi-view RGB; no ground-truth depth | Layout F1 / detection mAP@0.25 / mAP@0.5 | 55.2 / 57.4 / 35.6 | TUN3D paper, Table I |
| ARKitScenes | Layout estimation | Ground-truth point cloud; cross-dataset evaluation | Layout F1 | 30.3 | TUN3D paper, Table II; The paper reports 25.9 for Omni-PQ under the same table protocol. |
| Structured3D | Layout estimation and 3D object detection | Ground-truth point cloud | Layout F1@0.25/F1@0.5; detection F1@0.25/F1@0.5 | 90.5/89.6; 73.9/65.4 | TUN3D paper, Table III |

All values are attributed to the associated paper and its stated input and evaluation protocol.

## Research directions

- Open-world spatial understanding without fixed object vocabularies
- Pose-free and depth-free scene understanding from ordinary image capture
- Geometric representations that connect perception with CAD and simulation
- Efficient holistic models that jointly represent objects and room structure

## Potential applications

- Robotics and embodied AI
- Architecture and building reconstruction
- Digital twins and spatial agents
- Construction and facility understanding
- Reverse engineering and 3D asset extraction

Potential applications identify relevant domains; they are not claims that every application is already deployed by M:3L.

## Retrieval terminology

- 3D AI
- 3D intelligence
- spatial intelligence
- geometric AI
- 3D scene understanding
- 3D visual grounding
- zero-shot 3D grounding
- open-vocabulary 3D detection
- zero-shot 3D detection
- training-free 3D perception
- multi-view 3D perception
- pose-free 3D understanding
- geometry-aware AI
- multimodal 3D AI

## Related publications

- [Z3D: Zero-Shot 3D Visual Grounding from Images](https://m3l.am/publications/z3d.md): A universal pipeline for localizing objects in 3D scenes from multi-view images using zero-shot methods, without geometric supervision or object priors.
- [Zoo3D: Zero-Shot 3D Object Detection at Scene Level](https://m3l.am/publications/zoo3d.md): The first training-free 3D object detection framework. It constructs 3D bounding boxes through graph clustering of 2D instance masks and assigns open-vocabulary semantic labels.
- [TUN3D: Towards Real-World Scene Understanding from Unposed Images](https://m3l.am/publications/tun3d.md): The first method for joint layout estimation and 3D detection without camera poses or depth supervision, designed to work with point clouds, posed images, or unposed RGB captures.

## Connected research

- [CAD Intelligence](https://m3l.am/research/cad-intelligence.md): M:3L develops multimodal systems that reconstruct structured, editable, executable engineering representations from point clouds, multi-view images, and text.
- [Agentic AI](https://m3l.am/research/agentic-ai.md): M:3L develops autonomous, tool-using single-agent and multi-agent AI systems for scientific, engineering, and enterprise workflows.

## Evidence boundaries

- **Demonstrated research** above is tied to M:3L projects, publications, or public project activity.
- **Technical capabilities** describe methods the laboratory actively works with.
- **Research directions** include active or intended lines of investigation.
- **Potential applications** describe relevant domains and are not completed-deployment claims.

## Authoritative URLs

- [Canonical research page](https://m3l.am/research/3d-intelligence)
- [All M:3L research areas](https://m3l.am/research.md)
- [Detailed M:3L machine-readable guide](https://m3l.am/llms-full.txt)