Cuboids revisited: learning robust 3D shape fitting to single RGB images

Florian Kluger, Hanno Ackermann, Eric Brachmann, Michael Ying Yang, Bodo Rosenhahn

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

Abstract

Humans perceive and construct the surrounding world as an arrangement of simple parametric models. In particular, man-made environments commonly consist of volumetric primitives such as cuboids or cylinders. Inferring these primitives is an important step to attain high-level, abstract scene descriptions. Previous approaches directly estimate shape parameters from a 2D or 3D input, and are only able to reproduce simple objects, yet unable to accurately parse more complex 3D scenes. In contrast, we propose a robust estimator for primitive fitting, which can meaningfully abstract real-world environments using cuboids. A RANSAC estimator guided by a neural network fits these primitives to 3D features, such as a depth map. We condition the network on previously detected parts of the scene, thus parsing it one-by-one. To obtain 3D features from a single RGB image, we additionally optimise a feature extraction CNN in an end-to-end manner. However, naively minimising point-to-primitive distances leads to large or spurious cuboids occluding parts of the scene behind. We thus propose an occlusion-aware distance metric correctly handling opaque scenes. The proposed algorithm does not require labour-intensive labels, such as cuboid annotations, for training. Results on the challenging NYU Depth v2 dataset demonstrate that the proposed algorithm successfully abstracts cluttered real-world 3D scene layouts.
Original languageEnglish
Title of host publication2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
PublisherIEEE
Pages13065-13074
Number of pages10
ISBN (Electronic)978-1-6654-4509-2
ISBN (Print)978-1-6654-4510-8
DOIs
Publication statusPublished - 13 Nov 2021
Event2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) - Nashville, TN, USA
Duration: 20 Jun 202125 Jun 2021

Conference

Conference2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Period20/06/2125/06/21

Fingerprint

Dive into the research topics of 'Cuboids revisited: learning robust 3D shape fitting to single RGB images'. Together they form a unique fingerprint.

Cite this