Real-time Virtual-Try-On from a Single Example Image through Deep Inverse Graphics and Learned Differentiable Renderers

Kips, Robin; Jiang, Ruowei; Ba, Sileye; Duke, Brendan; Perrot, Matthieu; Gori, Pietro; Bloch, Isabelle

Real-time Virtual-Try-On from a Single Example Image through Deep Inverse Graphics and Learned Differentiable Renderers

Files

v41i2pp029-040.pdf (13.68 MB)

hair_video_examples.mp4 (52.91 MB)

lipstick_video_examples.mp4 (57.24 MB)

Date

2022

Authors

Kips, Robin
Jiang, Ruowei
Ba, Sileye
Duke, Brendan
Perrot, Matthieu
Gori, Pietro
Bloch, Isabelle

Publisher

The Eurographics Association and John Wiley & Sons Ltd.

Abstract

Augmented reality applications have rapidly spread across online retail platforms and social media, allowing consumers to virtually try-on a large variety of products, such as makeup, hair dying, or shoes. However, parametrizing a renderer to synthesize realistic images of a given product remains a challenging task that requires expert knowledge. While recent work has introduced neural rendering methods for virtual try-on from example images, current approaches are based on large generative models that cannot be used in real-time on mobile devices. This calls for a hybrid method that combines the advantages of computer graphics and neural rendering approaches. In this paper, we propose a novel framework based on deep learning to build a real-time inverse graphics encoder that learns to map a single example image into the parameter space of a given augmented reality rendering engine. Our method leverages self-supervised learning and does not require labeled training data, which makes it extendable to many virtual try-on applications. Furthermore, most augmented reality renderers are not differentiable in practice due to algorithmic choices or implementation constraints to reach real-time on portable devices. To relax the need for a graphics-based differentiable renderer in inverse graphics problems, we introduce a trainable imitator module. Our imitator is a generative network that learns to accurately reproduce the behavior of a given non-differentiable renderer. We propose a novel rendering sensitivity loss to train the imitator, which ensures that the network learns an accurate and continuous representation for each rendering parameter. Automatically learning a differentiable renderer, as proposed here, could be beneficial for various inverse graphics tasks. Our framework enables novel applications where consumers can virtually try-on a novel unknown product from an inspirational reference image on social media. It can also be used by computer graphics artists to automatically create realistic rendering from a reference product image.

CCS Concepts: Computing methodologies --> Computer vision; Machine learning; Computer graphics

        @article{10.1111:cgf.14456
,
journal = {Computer Graphics Forum},
title = {{Real-time Virtual-Try-On from a Single Example Image through Deep Inverse Graphics and Learned Differentiable Renderers
}},
author = {Kips, Robin and 
Jiang, Ruowei and 
Ba, Sileye and 
Duke, Brendan and 
Perrot, Matthieu and 
Gori, Pietro and 
Bloch, Isabelle
},
year = {2022
},
publisher = {The Eurographics Association and John Wiley & Sons Ltd.
},
ISSN = {1467-8659
},
DOI = {10.1111/cgf.14456
}
}