43-Issue 7

Permanent URI for this collection

https://diglib.eg.org/handle/10.2312/3607047

Browse

Now showing 1 - 20 of 57

Adversarial Unsupervised Domain Adaptation for 3D Semantic Segmentation with 2D Image Fusion of Dense Depth
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Zhang, Xindan; Li, Ying; Sheng, Huankun; Zhang, Xinnian; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Unsupervised domain adaptation (UDA) is increasingly used for 3D point cloud semantic segmentation tasks due to its ability to address the issue of missing labels for new domains. However, most existing unsupervised domain adaptation methods focus only on uni-modal data and are rarely applied to multi-modal data. Therefore, we propose a cross-modal UDA on multimodal datasets that contain 3D point clouds and 2D images for 3D Semantic Segmentation. Specifically, we first propose a Dual discriminator-based Domain Adaptation (Dd-bDA) module to enhance the adaptability of different domains. Second, given that the robustness of depth information to domain shifts can provide more details for semantic segmentation, we further employ a Dense depth Feature Fusion (DdFF) module to extract image features with rich depth cues. We evaluate our model in four unsupervised domain adaptation scenarios, i.e., dataset-to-dataset (A2D2→SemanticKITTI), Day-to-Night, country-tocountry (USA→Singapore), and synthetic-to-real (VirtualKITTI→SemanticKITTI). In all settings, the experimental results achieve significant improvements and surpass state-of-the-art models.
Anisotropic Specular Image-Based Lighting Based on BRDF Major Axis Sampling
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Cocco, Giovanni; Zanni, Cédric; Chermain, Xavier; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Anisotropic specular appearances are ubiquitous in the environment: brushed stainless steel pans, kettles, elevator walls, fur, or scratched plastics. Real-time rendering of these materials with image-based lighting is challenging due to the complex shape of the bidirectional reflectance distribution function (BRDF). We propose an anisotropic specular image-based lighting method that can serve as a drop-in replacement for the standard bent normal technique [Rev11]. Our method yields more realistic results with a 50% increase in computation time of the previous technique, using the same high dynamic range (HDR) preintegrated environment image. We use several environment samples positioned along the major axis of the specular microfacet BRDF. We derive an analytic formula to determine the two closest and two farthest points from the reflected direction on an approximation of the BRDF confidence region boundary. The two farthest points define the BRDF major axis, while the two closest points are used to approximate the BRDF width. The environment level of detail is derived from the BRDF width and the distance between the samples. We extensively compare our method with the bent normal technique and the ground truth using the GGX specular BRDF.
Cinematic Gaussians: Real-Time HDR Radiance Fields with Depth of Field
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Wang, Chao; Wolski, Krzysztof; Kerbl, Bernhard; Serrano, Ana; Bemama, Mojtaba; Seidel, Hans-Peter; Myszkowski, Karol; Leimkühler, Thomas; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Radiance field methods represent the state of the art in reconstructing complex scenes from multi-view photos. However, these reconstructions often suffer from one or both of the following limitations: First, they typically represent scenes in low dynamic range (LDR), which restricts their use to evenly lit environments and hinders immersive viewing experiences. Secondly, their reliance on a pinhole camera model, assuming all scene elements are in focus in the input images, presents practical challenges and complicates refocusing during novel-view synthesis. Addressing these limitations, we present a lightweight method based on 3D Gaussian Splatting that utilizes multi-view LDR images of a scene with varying exposure times, apertures, and focus distances as input to reconstruct a high-dynamic-range (HDR) radiance field. By incorporating analytical convolutions of Gaussians based on a thin-lens camera model as well as a tonemapping module, our reconstructions enable the rendering of HDR content with flexible refocusing capabilities. We demonstrate that our combined treatment of HDR and depth of field facilitates real-time cinematic rendering, outperforming the state of the art.
Color-Accurate Camera Capture with Multispectral Illumination and Multiple Exposures
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Gao, Hongyun; Mantiuk, Rafal K.; Finlayson, Graham D.; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Cameras cannot capture the same colors as those seen by the human eye because the eye and the cameras' sensors differ in their spectral sensitivity. To obtain a plausible approximation of perceived colors, the camera's Image Signal Processor (ISP) employs a color correction step. However, even advanced color correction methods cannot solve this underdetermined problem, and visible color inaccuracies are always present. Here, we explore an approach in which we can capture accurate colors with a regular camera by optimizing the spectral composition of the illuminant and capturing one or more exposures. We jointly optimize for the signal-to-noise ratio and for the color accuracy irrespective of the spectral composition of the scene. One or more images captured under controlled multispectral illuminants are then converted into a color-accurate image as seen under the standard illuminant of D65. Our optimization allows us to reduce the color error by 20-60% (in terms of CIEDE 2000), depending on the number of exposures and camera type. The method can be used in applications in which illumination can be controlled, and high colour accuracy is required, such as product photography or with a multispectral camera flash. The code is available at https://github.com/gfxdisp/multispectral_color_correction.
Controllable Anime Image Editing Based on the Probability of Attribute Tags
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Song, Zhenghao; Mo, Haoran; Gao, Chengying; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Editing anime images via probabilities of attribute tags allows controlling the degree of the manipulation in an intuitive and convenient manner. Existing methods fall short in the progressive modification and preservation of unintended regions in the input image. We propose a controllable anime image editing framework based on adjusting the tag probabilities, in which a probability encoding network (PEN) is developed to encode the probabilities into features that capture continuous characteristic of the probabilities. Thus, the encoded features are able to direct the generative process of a pre-trained diffusion model and facilitate the linear manipulation.We also introduce a local editing module that automatically identifies the intended regions and constrains the edits to be applied to those regions only, which preserves the others unchanged. Comprehensive comparisons with existing methods indicate the effectiveness of our framework in both one-shot and linear editing modes. Results in additional applications further demonstrate the generalization ability of our approach.
CoupNeRF: Property-aware Neural Radiance Fields for Multi-Material Coupled Scenario Reconstruction
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Li, Jin; Gao, Yang; Song, Wenfeng; Li, Yacong; Li, Shuai; Hao, Aimin; Qin, Hong; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Neural Radiance Fields (NeRFs) have achieved significant recognition for their proficiency in scene reconstruction and rendering by utilizing neural networks to depict intricate volumetric environments. Despite considerable research dedicated to reconstructing physical scenes, rare works succeed in challenging scenarios involving dynamic, multi-material objects. To alleviate, we introduce CoupNeRF, an efficient neural network architecture that is aware of multiple material properties. This architecture combines physically grounded continuum mechanics with NeRF, facilitating the identification of motion systems across a wide range of physical coupling scenarios. We first reconstruct specific-material of objects within 3D physical fields to learn material parameters. Then, we develop a method to model the neighbouring particles, enhancing the learning process specifically in regions where material transitions occur. The effectiveness of CoupNeRF is demonstrated through extensive experiments, showcasing its proficiency in accurately coupling and identifying the behavior of complex physical scenes that span multiple physics domains.
CrystalNet: Texture-Aware Neural Refraction Baking for Global Illumination
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Zhang, Ziyang; Simo-Serra, Edgar; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Neural rendering bakes global illumination and other computationally costly effects into the weights of a neural network, allowing to efficiently synthesize photorealistic images without relying on path tracing. In neural rendering approaches, G-buffers obtained from rasterization through direct rendering provide information regarding the scene such as position, normal, and textures to the neural network, achieving accurate and stable rendering quality in real-time. However, due to the use of G-buffers, existing methods struggle to accurately render transparency and refraction effects, as G-buffers do not capture any ray information from multiple light ray bounces. This limitation results in blurriness, distortions, and loss of detail in rendered images that contain transparency and refraction, and is particularly notable in scenes with refracted objects that have high-frequency textures. In this work, we propose a neural network architecture to encode critical rendering information, including texture coordinates from refracted rays, and enable reconstruction of high-frequency textures in areas with refraction. Our approach is able to achieve accurate refraction rendering in challenging scenes with a diversity of overlapping transparent objects. Experimental results demonstrate that our method can interactively render high quality refraction effects with global illumination, unlike existing neural rendering approaches. Our code can be found at https://github.com/ziyangz5/CrystalNet
Curved Image Triangulation Based on Differentiable Rendering
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Wang, Wanyi; Chen, Zhonggui; Fang, Lincong; Cao, Juan; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Image triangulation methods, which decompose an image into a series of triangles, are fundamental in artistic creation and image processing. This paper introduces a novel framework that integrates cubic Bézier curves into image triangulation, enabling the precise reconstruction of curved image features. Our developed framework constructs a well-structured curved triangle mesh, effectively preventing overlaps between curves. A refined energy function, grounded in differentiable rendering, establishes a direct link between mesh geometry and rendering effects and is instrumental in guiding the curved mesh generation. Additionally, we derive an explicit gradient formula with respect to mesh parameters, facilitating the adaptive and efficient optimization of these parameters to fully leverage the capabilities of cubic Bézier curves. Through experimental and comparative analyses with state-of-the-art methods, our approach demonstrates a significant enhancement in both numerical accuracy and visual quality.
CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Xiao, Chufeng; Fu, Hongbo; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Personalization techniques for large text-to-image (T2I) models allow users to incorporate new concepts from reference images. However, existing methods primarily rely on textual descriptions, leading to limited control over customized images and failing to support fine-grained and local editing (e.g., shape, pose, and details). In this paper, we identify sketches as an intuitive and versatile representation that can facilitate such control, e.g., contour lines capturing shape information and flow lines representing texture. This motivates us to explore a novel task of sketch concept extraction: given one or more sketch-image pairs, we aim to extract a special sketch concept that bridges the correspondence between the images and sketches, thus enabling sketch-based image synthesis and editing at a fine-grained level. To accomplish this, we introduce CustomSketching, a two-stage framework for extracting novel sketch concepts via few-shot learning. Considering that an object can often be depicted by a contour for general shapes and additional strokes for internal details, we introduce a dual-sketch representation to reduce the inherent ambiguity in sketch depiction. We employ a shape loss and a regularization loss to balance fidelity and editability during optimization. Through extensive experiments, a user study, and several applications, we show our method is effective and superior to the adapted baselines.
Density-Aware Diffusion Model for Efficient Image Dehazing
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Zhang, Ling; Bai, Wenxu; Xiao, Chunxia; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Existing image dehazing methods have made remarkable progress. However, they generally perform poorly on images with dense haze, and often suffer from unsatisfactory results with detail degradation or color distortion. In this paper, we propose a density-aware diffusion model (DADM) for image dehazing. Guided by the haze density, our DADM can handle images with dense haze and complex environments. Specifically, we introduce a density-aware dehazing network (DADNet) in the reverse diffusion process, which can help DADM gradually recover a clear haze-free image from a haze image. To improve the performance of the network, we design a cross-feature density extraction module (CDEModule) to extract the haze density for the image and a density-guided feature fusion block (DFFBlock) to learn the effective contextual features. Furthermore, we introduce an indirect sampling strategy in the test sampling process, which not only suppresses the accumulation of errors but also ensures the stability of the results. Extensive experiments on popular benchmarks validate the superior performance of the proposed method. The code is released in https://github.com/benchacha/DADM.
DiffPop: Plausibility-Guided Object Placement Diffusion for Image Composition
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Liu, Jiacheng; Zhou, Hang; Wei, Shida; Ma, Rui; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
In this paper, we address the problem of plausible object placement for the challenging task of realistic image composition. We propose DiffPop, the first framework that utilizes plausibility-guided denoising diffusion probabilistic model to learn the scale and spatial relations among multiple objects and the corresponding scene image. First, we train an unguided diffusion model to directly learn the object placement parameters in a self-supervised manner. Then, we develop a human-in-the-loop pipeline which exploits human labeling on the diffusion-generated composite images to provide the weak supervision for training a structural plausibility classifier. The classifier is further used to guide the diffusion sampling process towards generating the plausible object placement. Experimental results verify the superiority of our method for producing plausible and diverse composite images on the new Cityscapes-OP dataset and the public OPA dataset, as well as demonstrate its potential in applications such as data augmentation and multi-object placement tasks. Our dataset and code will be released.
Digital Garment Alteration
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Eggler, Anna Maria; Falque, Raphael; Liu, Mark; Vidal-Calleja, Teresa; Sorkine-Hornung, Olga; Pietroni, Nico; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Garment alteration is a practical technique to adapt an existing garment to fit a target body shape. Typically executed by skilled tailors, this process involves a series of strategic fabric operations-removing or adding material-to achieve the desired fit on a target body. We propose an innovative approach to automate this process by computing a set of practically feasible modifications that adapt an existing garment to fit a different body shape. We first assess the garment's fit on a reference body; then, we replicate this fit on the target by deriving a set of pattern modifications via a linear program. We compute these alterations by employing an iterative process that alternates between global geometric optimization and physical simulation. Our method utilizes geometry-based simulation of woven fabric's anisotropic behavior, accounts for tailoring details like seam matching, and incorporates elements such as darts or gussets. We validate our technique by producing digital and physical garments, demonstrating practical and achievable alterations.
Disentangled Lifespan Synthesis via Transformer-Based Nonlinear Regression
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Li, Mingyuan; Guo, Yingchun; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Lifespan face age transformation aims to generate facial images that accurately depict an individual's appearance at different age stages. This task is highly challenging due to the need for reasonable changes in facial features while preserving identity characteristics. Existing methods tend to synthesize unsatisfactory results, such as entangled facial attributes and low identity preservation, especially when dealing with large age gaps. Furthermore, over-manipulating the style vector may deviate it from the latent space and damage image quality. To address these issues, this paper introduces a novel nonlinear regression model- Disentangled Lifespan face Aging (DL-Aging) to achieve high-quality age transformation images. Specifically, we propose an age modulation encoder to extract age-related multi-scale facial features as key and value, and use the reconstructed style vector of the image as the query. The multi-head cross-attention in the W+ space is utilized to update the query for aging image reconstruction iteratively. This nonlinear transformation enables the model to learn a more disentangled mode of transformation, which is crucial for alleviating facial attribute entanglement. Additionally, we introduce a W+ space age regularization term to prevent excessive manipulation of the style vector and ensure it remains within theW+ space during transformation, thereby improving generation quality and aging accuracy. Extensive qualitative and quantitative experiments demonstrate that the proposed DL-Aging outperforms state-of-the-art methods regarding aging accuracy, image quality, attribute disentanglement, and identity preservation, especially for large age gaps.
Disk B-spline on S2: A Skeleton-based Region Representation on 2-Sphere
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Zheng, Chunhao; Zhao, Yuming; Wu, Zhongke; Wang, Xingce; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Due to the widespread applications of 2-dimensional spherical designs, there has been an increasing requirement of modeling on the S2 manifold in recent years. Due to the non-Euclidean nature of the sphere, it has some challenges to find a method to represent 2D regions on S2 manifold. In this paper, a skeleton-based representation method of regions on S2, disk B-spline(DBSC) on S2 is proposed. Firstly, we give the definition and basic algorithms of DBSC on S2. Then we provide the calculation method of DBSC on S2, which includes calculating the boundary points, internal points and their corresponding derivatives. Based on that, we give some modeling methods of DBSC on S2, including approximation, deformation. In the end, some stunning application examples of DBSC on S2 are shown. This work lays a theoretical foundation for further applications of DBSC on S2.
Distinguishing Structures from Textures by Patch-based Contrasts around Pixels for High-quality and Efficient Texture filtering
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Wang, Shengchun; Xu, Panpan; Hou, Fei; Wang, Wencheng; Zhao, Chong; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
It is still challenging with existing methods to distinguish structures from texture details, and so preventing texture filtering. Considering that the textures on both sides of a structural edge always differ much from each other in appearances, we determine whether a pixel is on a structure edge by exploiting the appearance contrast between patches around the pixel, and further propose an efficient implementation method. We demonstrate that our proposed method is more effective than existing methods to distinguish structures from texture details, and our required patches for texture measurement can be smaller than the used patches in existing methods by at least half. Thus, we can improve texture filtering on both quality and efficiency, as shown by the experimental results, e.g., we can handle the textured images with a resolution of 800 × 600 pixels in real-time. (The code is available at https://github.com/hefengxiyulu/MLPC)
DSGI-Net: Density-based Selective Grouping Point Cloud Learning Network for Indoor Scene
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Wen, Xin; Duan, Yao; Xu, Kai; Zhu, Chenyang; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Indoor scene point clouds exhibit diverse distributions and varying levels of sparsity, characterized by more intricate geometry and occlusion compared to outdoor scenes or individual objects. Despite recent advancements in 3D point cloud analysis introducing various network architectures, there remains a lack of frameworks tailored to the unique attributes of indoor scenarios. To address this, we propose DSGI-Net, a novel indoor scene point cloud learning network that can be integrated into existing models. The key innovation of this work is selectively grouping more informative neighbor points in sparse regions and promoting semantic consistency of the local area where different instances are in proximity but belong to distinct categories. Furthermore, our method encodes both semantic and spatial relationships between points in local regions to reduce the loss of local geometric details. Extensive experiments on the ScanNetv2, SUN RGB-D, and S3DIS indoor scene benchmarks demonstrate that our method is straightforward yet effective.
Evolutive 3D Urban Data Representation through Timeline Design Space
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Gautier, Corentin Le Bihan; Delanoy, Johanna; Gesquière, Gilles; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Cities are constantly changing to adapt to new societal and environmental challenges. Understanding their evolution is thus essential to make informed decisions about their future. To capture these changes, cities are increasingly offering digital 3D snapshots of their territory over time. However, existing tools to visualise these data typically represent the city at a specific point in time, limiting a comprehensive analysis of its evolution. In this paper, we propose a new method for simultaneously visualising different versions of the city in a 3D space. We integrate the different versions of the city along a new way of 3D timeline that can take different shapes depending on the needs of the user and the dataset being visualised. We propose four different shapes of timelines and three ways to place the versions along it. Our method places the versions such that there is no visual overlap for the user by varying the parameters of the timelines, and offer options to ease the understanding of the scene by changing the orientation or scale of the versions. We evaluate our method on different datasets to demonstrate the advantages and limitations of the different shapes of timeline and provide recommendations so as to which shape to chose.
Exploring Fast and Flexible Zero-Shot Low-Light Image/Video Enhancement
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Han, Xianjun; Bao, Taoli; Yang, Hongyu; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Low-light image/video enhancement is a challenging task when images or video are captured under harsh lighting conditions. Existing methods mostly formulate this task as an image-to-image conversion task via supervised or unsupervised learning. However, such conversion methods require an extremely large amount of data for training, whether paired or unpaired. In addition, these methods are restricted to specific training data, making it difficult for the trained model to enhance other types of images or video. In this paper, we explore a novel, fast and flexible, zero-shot, low-light image or video enhancement framework. Without relying on prior training or relationships among neighboring frames, we are committed to estimating the illumination of the input image/frame by a well-designed network. The proposed zero-shot, low-light image/video enhancement architecture includes illumination estimation and residual correction modules. The network architecture is very concise and does not require any paired or unpaired data during training, which allows low-light enhancement to be performed with several simple iterations. Despite its simplicity, we show that the method is fast and generalizes well to diverse lighting conditions. Many experiments on various images and videos qualitatively and quantitatively demonstrate the advantages of our method over state-of-the-art methods.
Faster Ray Tracing through Hierarchy Cut Code
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Xiang, WeiLai; Liu, FengQi; Tan, Zaonan; Li, Dan; Xu, PengZhan; Liu, MeiZhi; Kou, QiLong; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
We propose a novel ray reordering technique designed to accelerate the ray tracing process by encoding and sorting rays prior to traversal. Our method, called ''hierarchy cut code'', involves encoding rays based on the cuts of the hierarchical acceleration structure, rather than relying solely on spatial coordinates. This approach allows for a more effective adaptation to the acceleration structure, resulting in a more reliable and efficient encoding outcome. Furthermore, our research identifies ''bounding drift'' as a major obstacle in achieving better acceleration effects using longer sorting keys in existing reordering methods. Fortunately, our hierarchy cut code successfully overcomes this issue, providing improved performance in ray tracing. Experimental results demonstrate the effectiveness of our approach, showing up to a 1.81 times faster secondary ray tracing compared to existing methods. These promising results highlight the potential for further enhancement in the acceleration effect of reordering techniques, warranting further exploration and research in this exciting field.
FastFlow: GPU Acceleration of Flow and Depression Routing for Landscape Simulation
(The Eurographics Association and John Wiley & Sons Ltd., 2024) Jain, Aryamaan; Kerbl, Bernhard; Gain, James; Finley, Brandon; Cordonnier, Guillaume; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Terrain analysis plays an important role in computer graphics, hydrology and geomorphology. In particular, analyzing the path of material flow over a terrain with consideration of local depressions is a precursor to many further tasks in erosion, river formation, and plant ecosystem simulation. For example, fluvial erosion simulation used in terrain modeling computes water discharge to repeatedly locate erosion channels for soil removal and transport. Despite its significance, traditional methods face performance constraints, limiting their broader applicability. In this paper, we propose a novel GPU flow routing algorithm that computes the water discharge in O(logn) iterations for a terrain with n vertices (assuming n processors). We also provide a depression routing algorithm to route the water out of local minima formed by depressions in the terrain, which converges in O(log2 n) iterations. Our implementation of these algorithms leads to a 5× speedup for flow routing and 34× to 52× speedup for depression routing compared to previous work on a 10242 terrain, enabling interactive control of terrain simulation.

Browse

Browsing 43-Issue 7 by Title

Results Per Page

Sort Options