HaLo-NeRF: Learning Geometry-Guided Semantics for Exploring Unconstrained Photo Collections

Simple item page
dc.contributor.authorDudai, Chenen_US
dc.contributor.authorAlper, Morrisen_US
dc.contributor.authorBezalel, Hanaen_US
dc.contributor.authorHanocka, Ranaen_US
dc.contributor.authorLang, Itaien_US
dc.contributor.authorAverbuch-Elor, Hadaren_US
dc.contributor.editorBermano, Amit H.en_US
dc.contributor.editorKalogerakis, Evangelosen_US
dc.date.accessioned2024-04-30T09:06:50Z
dc.date.available2024-04-30T09:06:50Z
dc.date.issued2024
dc.description.abstractInternet image collections containing photos captured by crowds of photographers show promise for enabling digital exploration of large-scale tourist landmarks. However, prior works focus primarily on geometric reconstruction and visualization, neglecting the key role of language in providing a semantic interface for navigation and fine-grained understanding. In more constrained 3D domains, recent methods have leveraged modern vision-and-language models as a strong prior of 2D visual semantics. While these models display an excellent understanding of broad visual semantics, they struggle with unconstrained photo collections depicting such tourist landmarks, as they lack expert knowledge of the architectural domain and fail to exploit the geometric consistency of images capturing multiple views of such scenes. In this work, we present a localization system that connects neural representations of scenes depicting large-scale landmarks with text describing a semantic region within the scene, by harnessing the power of SOTA vision-and-language models with adaptations for understanding landmark scene semantics. To bolster such models with fine-grained knowledge, we leverage large-scale Internet data containing images of similar landmarks along with weakly-related textual information. Our approach is built upon the premise that images physically grounded in space can provide a powerful supervision signal for localizing new concepts, whose semantics may be unlocked from Internet textual metadata with large language models. We use correspondences between views of scenes to bootstrap spatial understanding of these semantics, providing guidance for 3D-compatible segmentation that ultimately lifts to a volumetric scene representation. To evaluate our method, we present a new benchmark dataset containing large-scale scenes with groundtruth segmentations for multiple semantic concepts. Our results show that HaLo-NeRF can accurately localize a variety of semantic concepts related to architectural landmarks, surpassing the results of other 3D models as well as strong 2D segmentation baselines. Our code and data are publicly available at https://tau-vailab.github.io/HaLo-NeRF/.en_US
dc.description.number2
dc.description.sectionheadersShape and Scene Understanding
dc.description.seriesinformationComputer Graphics Forum
dc.description.volume43
dc.identifier.doi10.1111/cgf.15006
dc.identifier.issn1467-8659
dc.identifier.pages12 pages
dc.identifier.urihttps://doi.org/10.1111/cgf.15006
dc.identifier.urihttps://diglib.eg.org/handle/10.1111/cgf15006
dc.publisherThe Eurographics Association and John Wiley & Sons Ltd.en_US
dc.rightsAttribution 4.0 International License
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.subjectCCS Concepts: Computing methodologies -> 3D imaging; Rendering; Image segmentation
dc.subjectComputing methodologies
dc.subject3D imaging
dc.subjectRendering
dc.subjectImage segmentation
dc.titleHaLo-NeRF: Learning Geometry-Guided Semantics for Exploring Unconstrained Photo Collectionsen_US
Files
Original bundle
Now showing 1 - 2 of 2
No Thumbnail Available
Name:
v43i2_02_15006.pdf
Size:
24.3 MB
Format:
Adobe Portable Document Format
Download
No Thumbnail Available
Name:
supplementary.zip
Size:
35.98 MB
Format:
Zip file
Download
Collections
43-Issue 2