SS-SfP: Neural Inverse Rendering for Self Supervised Shape from (Mixed) Polarization

Tiwari, Ashish; Raman, Shanmuganathan

SS-SfP: Neural Inverse Rendering for Self Supervised Shape from (Mixed) Polarization

Files

001-009.pdf (50.91 MB)

paper1022_mm.pdf (26.51 MB)

Date

2023

Authors

Tiwari, Ashish
Raman, Shanmuganathan

Publisher

The Eurographics Association

Abstract

We present a novel inverse rendering-based framework to estimate the 3D shape (per-pixel surface normals and depth) of objects and scenes from single-view polarization images, the problem popularly known as Shape from Polarization (SfP). The existing physics-based and learning-based methods for SfP perform under certain restrictions, i.e., (a) purely diffuse or purely specular reflections, which are seldom in the real surfaces, (b) availability of the ground truth surface normals for direct supervision that are hard to acquire and are limited by the scanner's resolution, and (c) known refractive index. To overcome these restrictions, we start by learning to separate the partially-polarized diffuse and specular reflection components, which we call reflectance cues, based on a modified polarization reflection model and then estimate shape under mixed polarization through an inverse-rendering based self-supervised deep learning framework called SS-SfP, guided by the polarization data and estimated reflectance cues. Furthermore, we also obtain the refractive index as a non-linear least squares solution. Through extensive quantitative and qualitative evaluation, we establish the efficacy of the proposed framework over simple single-object scenes from DeepSfP dataset and complex in-the-wild scenes from SPW dataset in an entirely self-supervised setting. To the best of our knowledge, this is the first learning-based approach to address SfP under mixed polarization in a completely selfsupervised framework. Code will be made publicly available.

CCS Concepts: Computing methodologies -> Computer Vision; Image-based Rendering

        @inproceedings{10.2312:pg.20231265
,
booktitle = {Pacific Graphics Short Papers and Posters
},
editor = {Chaine, Raphaëlle and 
Deng, Zhigang and 
Kim, Min H.
},
title = {{SS-SfP: Neural Inverse Rendering for Self Supervised Shape from (Mixed) Polarization
}},
author = {Tiwari, Ashish and 
Raman, Shanmuganathan
},
year = {2023
},
publisher = {The Eurographics Association
},
ISBN = {978-3-03868-234-9
},
DOI = {10.2312/pg.20231265
}
}

URI

https://doi.org/10.2312/pg.20231265
https://diglib.eg.org:443/handle/10.2312/pg20231265

Collections

PG2023 Short Papers and Posters

Full item page