| tags: | |
| - 3d-object-detection | |
| - open-vocabulary | |
| - point-cloud | |
| datasets: | |
| - lvis | |
| - sunrgbd | |
| - scannet | |
| pipeline_tag: object-detection | |
| # ImOV3D: Learning Open-Vocabulary Point Clouds 3D Object Detection from Only 2D Images | |
| **NeurIPS 2024** | [Paper](https://arxiv.org/abs/2410.24001) | [Project Page](https://yangtiming.github.io/ImOV3D_Page/) | [Code](https://github.com/yangtiming/ImOV3D) | |
| > Timing Yang\*, Yuanliang Ju\*, Li Yi | |
| > Shanghai Qi Zhi Institute, IIIS Tsinghua University, Shanghai AI Lab | |
| ## Overview | |
| ImOV3D is the **first open-vocabulary 3D object detector trained entirely from 2D images** — no 3D ground truth required. It bridges the 2D-3D modality gap via flexible modality conversion: lifting 2D images to pseudo point clouds (monocular depth estimation) and rendering point clouds back to pseudo images (ControlNet). This creates a unified image-PC representation for training a multimodal 3D detector. | |
| ## Citation | |
| ```bibtex | |
| @article{yang2024imov3d, | |
| title={ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images}, | |
| author={Yang, Timing and Ju, Yuanliang and Yi, Li}, | |
| journal={Advances in Neural Information Processing Systems}, | |
| volume={37}, | |
| pages={141261--141291}, | |
| year={2024} | |
| } | |
| ``` | |
| ## Contact | |
| Timing Yang: timingya@usc.edu · Yuanliang Ju: yuanliang.ju@mail.utoronto.ca | |