Path from Photorealism to Perceptual Realism
No Thumbnail Available
Date
2022-09
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Photorealism in computer graphics --- rendering images that appear as realistic as photographs --- has matured to the point that it is now widely used in industry. With emerging 3D display technologies, the next big challenge in graphics is to achieve Perceptual Realism --- producing virtual imagery that is perceptually indistinguishable from real-world 3D scenes. Such a significant upgrade in the level of realism offers highly immersive and engaging experiences that have the potential to revolutionise numerous aspects of life and society, including entertainment, social connections, education, business, scientific research, engineering, and design.
While perceptual realism puts strict requirements on the quality of reproduction, the virtual scene does not have to be identical in light distributions to its physical counterpart to be perceptually realistic, providing that it is visually indistinguishable to human eyes. Due to the limitations of human vision, a significant improvement in perceptual realism can, in principle, be achieved by fulfilling the essential visual requirements with sufficient qualities and without having to reconstruct the physically accurate distribution of light. In this dissertation, we start by discussing the capabilities and limits of the human visual system, which serves as a basis for the analysis of the essential visual requirements for perceptual realism. Next, we introduce a Perceptually Realistic Graphics (PRG) pipeline consisting of the acquisition, representation, and reproduction of the plenoptic function of a 3D scene. Finally, we demonstrate that taking advantage of the limits and mechanisms of the human visual system can significantly improve this pipeline.
Specifically, we present three approaches to push the quality of virtual imagery towards perceptual realism.
First, we introduce DiCE, a real-time rendering algorithm that exploits the binocular fusion mechanism of the human visual system to boost the perceived local contrast of stereoscopic displays. The method was inspired by an established model of binocular contrast fusion. To optimise the experience of binocular fusion, we proposed and empirically validated a rivalry-prediction model that better controls rivalry. Next, we introduce Dark Stereo, another real-time rendering algorithm that facilitates depth perception from binocular depth cues for stereoscopic displays, especially those under low luminance. The algorithm was designed based on a proposed model of stereo constancy that predicts the precision of binocular depth cues for a given contrast and luminance. Both DiCE and Dark Stereo have been experimentally demonstrated to be effective in improving realism. Their real-time performance also makes them readily integrable into any existing VR rendering pipeline. Nonetheless, only improving rendering is not sufficient to meet all the visual requirements for perceptual realism. The overall fidelity of a typical stereoscopic VR display is still confined by its limited dynamic range, low spatial resolution, optical aberrations, and vergence-accommodation conflicts.
To push the limits of the overall fidelity, we present a High-Dynamic-Range Multi-Focal Stereo display (HDRMFS display) with an end-to-end imaging and rendering system. The system can visually reproduce real-world 3D objects with high resolution, accurate colour, a wide dynamic range and contrast, and most depth cues, including binocular disparity and focal depth cues,
and permits a direct comparison between real and virtual scenes. It is the first work that achieves a close perceptual match between a physical 3D object and its virtual counterpart. The fidelity of reproduction has been confirmed by a Visual Turing Test (VTT) where naive participants failed to discern any difference between the real and virtual objects in more than half of the trials. The test provides insights to better understand the conditions necessary to achieve perceptual realism. In the long term, we foresee this system as a crucial step in the development of perceptually realistic graphics, for not only a quality unprecedentedly achieved but also a fundamental approach that can effectively identify bottlenecks and direct future studies for perceptually realistic graphics.
Description
Realism is a primary pursuit in computer graphics, particularly for Extended Reality (XR) applications, where realism is taken to a new level, transitioning from Photorealism to Perceptual Realism, which aims to create virtual scenes that are perceptually indistinguishable from real-world 3D scenes. This thesis delves into the fundamental question of achieving perceptual realism in extended reality. It presents the first work that established an end-to-end 3D imaging and display system that successfully passed a Visual Turing Test — In more than half of the trials, naive observers were unable to discern any difference between a physical 3D object and its virtually rendered counterpart. Passing a visual Turing test is considered the holy grail of perceptually realistic graphics. This dissertation marks a significant milestone that combines all different aspects of graphics towards the ultimate goal of digitising and visually reproducing a physical 3D scene. It demonstrates, for the first time, that perceptual realism can be achieved by fulfilling the key visual requirements rather than precisely replicating the physical distribution of light.