3D-Aware Image Relighting with Object Removal from Single Image

Abstract
We propose a method to relight scenes in a single image while removing unwanted objects by the combination of 3D-aware inpainting and relighting for a new functionality in image editing. First, the proposed method estimates the depth image from an RGB image using single-view depth estimation. Next, the RGB and depth images are masked by the user by specifying unwanted objects. Then, the masked RGB and depth images are simultaneously inpainted by our proposed neural network. For relighiting, a 3D mesh model is first reconstructed from the inpainted depth image, and is then relit with a standard relighting pipeline. In this process, removing cast shadows and sky areas and albedo estimation are optionally performed to suppress the artifacts in outdoor scenes. Through these processes, various types of relighting can be achieved from a single photograph while excluding the colors and shapes of unwanted objects.
Description

CCS Concepts: Computing methodologies -> Image processing; Human-centered computing -> Mixed / augmented reality

        
@inproceedings{
10.2312:egve.20221293
, booktitle = {
ICAT-EGVE 2022 - International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments - Posters and Demos
}, editor = {
Theophilus Teo
and
Ryota Kondo
}, title = {{
3D-Aware Image Relighting with Object Removal from Single Image
}}, author = {
Zhang, Yujia
and
Perusquia-Hernández, Monica
and
Isoyama, Naoya
and
Kawai, Norihiko
and
Uchiyama, Hideaki
and
Sakata, Nobuchika
and
Kiyokawa, Kiyoshi
}, year = {
2022
}, publisher = {
The Eurographics Association
}, ISSN = {
1727-530X
}, ISBN = {
978-3-03868-192-2
}, DOI = {
10.2312/egve.20221293
} }
Citation